esp@cenet document view 



Page 1 of 1 



Device mounted in vehicle 



Publication number: EP0893308 
Publication date: 1999-01-27 

Inventor: KUBOTA TOMOKI (JP): HORI KOJI (JP); MAZDA 

MANABU (JP); ADACHI KAZUHIDE (JP); MUKAI KOUJI 
(JP) 

Applicant: EQUOS RESEARCH KK (JP) 

aassificatlon; 

- intematianal: B60R16/02; B60R16/023; B60R16/037; G01C21/36; 

B60R16/02; B60R16/023; B60R16/037; G01C21/34; 
(IPC1-7): B60R16/02 

- European: B60R16/023D3; B60R16/037B; G01C21/36 
Application numben EP199801 13679 19980722 

Priority number(s): JP19970212491 19970722; JP1998006441 4 19980227; 

JP19980082626 19980312; JP19980082710 19980313; 
JP19980095386 19980323; JP19980095387 19980323 



Also published as: 

US6249720 (B1) 
EP0893308 (A3) 
EP0893308 (B1) 
DE69836846T(T2) 



cited documents: 

1 US5555502 
m JP9 102098 
1 JP3215891 



Report a data error here 



Abstract of EP0893308 
A device is mounted in a vehicle which allows 
one or more of personified agent or imaginary 
living body to appear before a driver or 
passengers In the vehicle for communication 
therewith. A current status sensor (40) detects 
predetermined vehicle current status data, and a 
memory (292, 293) stores study data in response 
to entry of the latest current status data. One of a 
plurality of communication programs stored in a 
program table (291 ) may be selected in 
accordance with the current status data and/or 
the study data. An agent control unit (11) controls 
activity of the agent in accordance with the 
selected communication program. The 
detemiined agent activity is outputted through a 
display (27) and a speaker (25) for 
communication with the driver. 
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Description 

[0001 ] This invention relates to a device mounted in a 
vehicle, and more particularly to a device for controlling 
activity of a personified agent on a display in a vehicle 
room for communication with a user such as a driver. 
[0002] Various audio instruments such as radio, cas- 
sette tape player, CD player, MD player, etc. are installed 
in a vehicle to provide comfort and relaxation to the driver 
while driving the vehicle. This is only one-way commu- 
nication. 

[0003] Bi-directional communication between the driv- 
er and his or her friends outside of the vehicle may be 
achieved by using radio communication medium such as 
radio transceiver and portable telephone. However, the 
driver have to find out a companion to taik to by operation 
of the radio communication medium, which might be dan- 
gerous during driving. The companion outside the vehicle 
can not understand the vehicle and driving condition that 
the driver meets with. 

[0004] tn the Japanese patent laid-open publication 
No. 9- 102 098 specific infomriation is transmitted to the 
driver. To make infonnation provided for a driver easier 
to understand and accurately transmit the information at 
a glance even during a vehicle travel by transmitting the 
infomriation to the driver through expressions of a human 
and visual expressions by motion, etc., when the vehicle 
infonnation is displayed on a display to infomri the driver 
of the state of the vehicle, explain how to operate a de- 
vice, or give a route guidance or travel warning. 
[0005] Thereto the Japanese patent-laid-open publi- 
cation No. 9- 1 02 098 provides a detecting means which 
detects the travel position, state, etc., of the vehicle is 
mounted on the vehicle body, and the device is equipped 
with a storage means which previously stores human 
actions and expressions used to guide the vehicle, wam 
a travel, and explain the state and a reproducing means 
which reproduces or puts the stored data together, when 
a guide signal or waming signal giving an important in- 
fluence on the vehicle travel is received from the detect- 
ing means , the expression, action, etc., of the human 
corresponding to the contents of the waming, etc., are 
reproduced by the reproducing means and displayed as 
image information on the display to transmit the easier- 
to-see accurate infonnation. 

[0006] In accordance with this prior art, the same ex- 
pression and/or activity of the agent is appearing in the 
same situation. Thus, this provides improved visual liter- 
acy but is still a kind of conventional meters. 
[0007] Accordingly, it is an object of the present inven- 
tion to provide a novel device mounted In a vehicle that 
provides bi-directional communication with the user in 
the vehicle. 

[0008] Another object of the present invention is to pro- 
vide bi-directional communication with the user in a ve- 
hicle by means of a personified agent whose activities 
are detennined not only by the current vehicle condition 
but also by the results of past record study. 



[0009] To achieve the above object, in accordance 
with the present invention, a device is mounted in a ve- 
hicle which allows one or more of personified agent or 
imaginary living body to appear before a driver or pas- 

5 sengers in the vehicle for communication therewith. 
[001 0] In accordance with an aspect of the present in- 
vention, the device comprising first sensor means for de- 
tecting predetennined current status; first memory 
means for storing study data renewable in response to 

10 entry of the latest curent status detected by the first sen- 
sor means; at least one personified agent communlcat- 
able with user in the vehicle; second memory means for 
storing a plurality of communication programs; commu- 
nication program select means for detemiining one of 

15 the communication programs to be executed in accord- 
ance with the cun-ent status detected by the first sensor 
means and/or the study data currently stored in the first 
memory means; agent control means for controlling ac- 
tivity of the agent in accondance with the selected com- 

20 munication program; agent output means for outputting 
the agenf s activity to the user in the vehicle determined 
by the agent control means. 

[0011] In a prefen-ed embodiment, the first sensor 
means detects at least one vehicle condition selected 

25 from the group consisting of a current vehicle position, a 
curent vehicle speed, a cun-ent shift position, winker op- 
eration, wiper operation, a coolant temperature, an en- 
gine oil temperature, an engine oil pressure, a passenger 
room temperature, an open air temperature, ignition 

30 switch operation, a brake pedal stroke, an accelerator 
pedal stroke, side brake operation, horn operation, belt 
loosening, a fuel remainder, light or lamp operation, seat 
belt operation, door opening, window opening, ABS op- 
eration, air-conditioner operation, distance to the preced- 

35 ing and succeeding vehicles and decrease of air pressure 
in tire. 

[0012] The user's own data may be included in the 
current status. The user's own data may also be consid- 
ered in determining the communication program to be 

40 executed. 

[001 3] The device of the present invention may further 
comprises second sensor means for detecting user's re- 
sponse to the agenf s activity outputted by the agent out- 
put means, and third memory means for storing user's 

45 response data detected by the second sensor means. In 
this embodiment, the user's response data is also con- 
sidered in detennining the communication program to be 
executed. The user's response data is subjected to study 
so that the study data in the first memory means is also 

50 renewable in response to entry of the latest user's re- 
sponse data. 

[0014] The agent output means may comprise one or 
both of display means and speaker means. In a preferred 
embodiment, utterance pattern of the agent outputted by 
55 the speaker means is selectively detennined by the agent 
control means. In another prefen^ed embodiment, ap- 
pearance pattem of the agent outputted by the display 
means is selectively detennined by the agent control 
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means. 

[0015] The device may Include agent's character se- 
lect means for determining a specific character to be giv- 
en to the agent. In this embodiment, the agent control 
means determines the agent's activity in consideration s 
of the agent's character determined by the agent's char- 
acter select means. 

[001 6] The device may include fourth memory means 
for storing data indicating user's familiarity to agent op- 
eration of the device. The agent control means deter- io 
mines the agent's activity in consideration of the agent's 
familiarity stored in the fourth memory means. 
[0017] In another prefen-ed embodiment, the agent 
output means allows a plurality of different agents to be 
distinguishably recognized one another by the user in '5 
the vehicle. For example, the device includes fifth mem- 
ory means for storing sub-agent program, which is se- 
lected In conrespondence to a specific communication 
program stored in the second memory means so that a 
sub-agent of the selected sub-agent program is output- 20 
ted by the agent output means together with the main 
agent of the executed communication program. The sub- 
agent may be a help-agent explaining the activity of the 
main agent. The sub-agent may be a short-cut agent 
allowing short-cut of operation executed by the main 25 
agent communication program. Preferably, the sub- 
agent program becomes operative when data indicating 
user's familiarity to agent operation of the device Is great- 
er than a predetermined critical value. 
[0018] In another preferred embodiment, a back- 30 
ground of the display means is selectively detemnlned so 
that the agent appears on the display means together 
with the selected background. 
[001 9] The above and other objects of the present In- 
vention will become apparent from the following descrip- 35 
tion when read in conjunction with the accompanying 
drawings In which: 

Fig. 1 Is a block diagram showing an agent system 
in accordance with a first embodiment of the present 40 
invention: Fig. 2 is an example of a program select 
table; Fig. 3 is a table describing agent's activities to 
be outputted when a specific program number is se- 
lected In reference to Fig. 2; Figs. 4A-4D represent 
a series of an agent's activity shown on a display by 
way of example; Fig. 5 is an example of a study item 
data in an agent data memory of Fig. 1 ; Figs. 6A and 
6B represent an example of response data in the 
agent data memory; Fig. 7 shows various data files 
stored in a navigation data memory; Fig. 8 shows so 
various sensors in a current status sensor; Fig. 9 is 
a flowchart of the agent control operation executed 
In this embodiment; Figs. 1 0A-IOC show an example 
of the agent activity outputted In this embodiment; 
Figs. 1 1 A-11 C show another example of the agent ss 
activity outputted in this embodiment; Fig. 12 shows 
contents of the agent data memory in a second em- 
bodiment; Rg. 1 3 is an example of an agent char- 



acter select table In the second embodiment; Fig. 14 
is an example of a voice output pattem select table 
in the second embodiment; Fig. 1 5 is a flowchart 
showing a main routine of the agent control operation 
In the second embodiment; Fig. 16 is a flowchart 
showing a sub-routine executed at step S1 1 4 In Fig. 
1 5, Fig. 1 7 shows contents of the agent data memory 
in another modified embodiment; Fig. 18 is a flow- 
chart showing the agent control operation to be car- 
ried out at S1 14 In Rg. 15; Figs. 19A-19C show an 
example of the agent activities outputted In this em- 
bodiment; Fig. 20A represents a example of a pro- 
gram select table; Fig. 20B represents the agenf s 
activities to be outputted in the communication pro- 
gram selected by the table of Fig. 20A; Fig. 21 shows 
contents of the agent data memory in still another 
modified embodiment; Fig. 22 shows an example of 
the familiarity data in the agent data of Fig. 21 ; Rg. 
23 shows contents of the agent data memory in still 
another modified embodiment; Fig. 24 Is an example 
of the help-agent select table in the agent data mem- 
ory of Fig. 23; Fig. 25 shows the help-agent activities 
to be outputted in the conresponding help-agent pro- 
gram number in the table of Fig. 24; Fig. 26 Is a 
flowchart of the main routine of the agent control op- 
eration In accordance with this embodiment; Figs. 
27A-27C illustrate an example of the agent control 
operation made by the flowchart of Fig. 26; Fig. 28 
shows contents of the agent data memory in still an- 
other modified embodiment; Fig. 29 Is an example 
of the sub-agent program table in the agent data 
memory of Fig. 28; Fig. 30 shows shortrcut agent 
activities to be outputted in the corresponding short- 
cut agent program number in the table of Rg. 29; 
Fig. 31 is an example of the contents of the frequent 
program data In the agent data memory of Fig. 28; 
Fig. 32 is a flowchart of the main routine of the agent 
control operation in accordance with this embodi- 
ment; Fig. 33 is an explanatory view showing multi- 
stage operation of the communication program for 
menu selection; Fig. 34 is an explanatory view show- 
ing the short-cut agent activity Incorporated into the 
multi-stage operation of Fig. 33; Fig. 35 is a flowchart 
of the short-cut agent operation that is a sub-routine 
branched from the main routine of Fig. 15; Fig. 36 
shows an example of the display when the short-cut 
agent appears thereon together with the main agent; 
Fig. 37 shows an example of the display when only 
the main agent appears thereon; Fig. 38 shows con- 
tents of the agent data memory in still another mod- 
ified embodiment; Fig. 39 shows an example of the 
study item data in the agent data memory of Fig. 38; 
Fig. 40 shows an example of the background select 
table in the agent data memory of Fig. 38; Rgs. 41 A- 
41 D show examples of the selected background that 
is shown on the display together with the prescribed 
agent's activity; Fig. 42 shows an example of the 
response recognition data in the agent data memory 
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of Fig. 38; Fig. 43 is a flowchart of the agent control 
operation to be executed in this embodiment as a 
sub-routine branched off from the main routine of 
Fig. 15; Figs. 44B and 44C show pictures succes- 
sively appearing on the display on the conditions of. 
Fig. 44A in this embodiment; Fig. 45 shows contents 
of the agent data memory in still another modified 
embodiment; Fig. 46 shows an example of the ap- 
pearance select table in the agent data memory in 
Fig. 45; Fig. 47 is a flowchart of the agent control 
operation to be executed in this embodiment as a 
sub-routine branched off from the main routine of 
Fig. 15; Figs. 48A-48C show an example of the 
agent's appearance and activity appearing on the 
display in this embodiment; Figs. 49A-49B show an- 
other example of the agent's appearance and activity 
in this embodiment; and Fig. 50 show still another 
example of the agenfs appearance and activity in a 
modified embodiment 

[0020] The first embodiment of the present Invention 
will be herein below described in more detail in reference 
to Figs. 1-11 of the accompanying drawings, 
[0021] In this embodiment, a personified agent ap- 
pears on a graphic display in a vehicle. The agent may 
be represented as a plane image or three-dimensional 
image by holography, for example. The activity (including 
utterance) of the agent is controlled in response to the 
vehicle condition and the result of past record study. The 
vehicle condition may include various condition of the 
vehicle itself, the driver, the passenger(s) and preceding 
and succeeding vehicles. The past record study means 
study of past record of the vehicle condition and may 
Include reaction of the user (driver) to the agent's activity 
appearing on the display. Thus, the driver may commu- 
nicates with the agent in the vehicle, even while driving. 
[0022] The "agent" used herein is defined as an imag- 
inary personified subject who can make judgement and 
study like a human body. Accordingly, the same agent 
makes different communication with the driver on the dif- 
ferent vehicle condition. Even on the same vehicle con- 
dition, the past record study may result in different com- 
munication by the agent. The agent Is not a supertiuman 
and may sometime make mis-judgement and blunder as 
fas as it does not affect the vehicle driving. Such mis- 
judgement and blunder made by the agent may be cor- 
rected in accordance with the driver's reaction, which is 
a kind of the past record study. 
[0023] The agent represented on a display may be like 
a human body. It may also be like an animal such as a 
chick, dog, cat, frog, rat, etc. It may also be a robot or 
any character. The agent may be a child at the beginning 
and is grown with time. The appearance and voice of the 
agent may be selected from predetermined plural ones. 
[0024] Fig. 1 1s a block diagram of the device embod- 
ying the present invention. There is a processing unit 1 
including a navigation processing unit 10 for finding out 
a drive route to a predetemriined goal that is known to 



the driver by visual and/or voice message, an agent 
processing unit 1 1 for controlling the activity of the agent, 
an interface (1/F) 1 2, an image processing unit 1 3. a voice 
control unit 1 4 and a cun-ent status data processing sec- 
5 tion15. 

[0025] Navigation processing unit 10 and agent 
processing unit 1 1 have respectively a central processing 
unit (CPU) that operates in response to the Input infor- 
mation to perfonn various data processing operation and 

10 control of parts. To CPU are connected ROM, RAM and 
timer through data bus lines. Navigation processing unit 
1 0 and agent processing unit 1 1 are connected with each 
other by a network so that data processed in one unit 
may be given to the other unit. ROM is a read-only-mem- 

15 ory storing various data and programs necessary to per- 
fonn control in CPU. RAM is a random-access-memory 
to be used as a worthing memory when CPU perfonns 
data processing operation. 

[0026] CPU of navigation processing unit 1 0 and agent 

20 processing unit 11 reads out the programs in ROM to 
conduct necessary operation. In modified embodiment, 
CPU reads out the programs from a memory medium In 
a memory drive 23, in which case the programs are then 
installed in an agent data memory 29, a navigation data 

25 memory 30, a hard disk (not shown) or any other memory. 
A necessary program is loaded from the memory to RAM 
for execution. It may be possible that CPU reads out the 
necessary program from the memory drive 23 to load it 
directly to RAM. 

30 [0027] To navigation processing unit 1 0 are connected 
a current position sensor 21 and navigation data memory 
30. To agent processing unit 1 1 1s connected agent data 
memory 29. An input device 22, memory drive 23 and a 
communication control device 24 are connected to l/F 

35 1 2. A display 27 and a camera 28 are connected to image 
processing unit 1 3. A voice processing unit 25 and a mi- 
crophone 26 are connected to voice control unit 14. A 
current status sensors 40 are connected to current status 
data processing section 15. 

40 [0028] Current position sensor21 for detecting the cur- 
rent absolute position (its latitude and longitude) of the 
vehicle has a GPS (global positioning system) receiver 
211. a direction sensor 212, a steering sensor 213, a 
distance sensor 214 and a beacon sensor 215 that re- 

45 ceives the position signal from a beacon located on the 
road. GPS receiver 21 1 receives radio waves from earth 
satellites to determine the vehicle position. Beacon sen- 
sor 21 5 receives positional infomriation from beacons ar- 
ranged on roads. GPS receiver 21 1 and beacon sensor 

50 21 5 can solely act to measure the vehicle position. Where 
GPS receiver 21 1 can not receive the radio waves and 
beacon sensor 215 can not receive the positional infor- 
mation, direction sensor 212 and distance sensor 214 
cooperates with each other to infer the current position. 

55 [0029] Direction sensor 21 2 includes a terrestrial mag- 
netism sensor that detects ten^estrial magnetism to de- 
temiine orientation of the vehicle, a gyromagnetic sensor 
that detects a rotational angle velocity of the vehicle that 
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is Integrated to determine orientation of tlie vehicle, a 
pair of wheel sensors that detect an output pulse differ- 
ence (a difference of movement between left and right 
wheels) to detect an angle of gyration of the vehicle. 
Steering sensor 213 employs an optical sensor or rota- 
tion resistance volume mounted to a rotating member of 
a steering to detect a steering angle. Distance sensor 
214 detects a distance of movement by detecting revo- 
lution of the wheels or acceleration, for example. 
[0030] Input device 22 is used to input any data, for 
example, the current position and the goal when starting 
navigation processing. Input device 22 also act as means 
for Inputting the driver's response to the agent's message 
or question. An example of input device 22 is a touch 
panel arranged on display 27 and adapted to input any 
Information by a touch on a l<ey or menu represented 
thereon. Another example of Input device 22 is a key- 
board, mouse, bar code reader, write pen, joy sticit. In- 
frared remote controllable input device and voice ana- 
lyzer. An infrared remote controllable input device may 
cooperate with a receiver that receives signals from the 
input device. The remote controllable input device has a 
joy stick for moving a cursor on a display section, menu 
designating keys or buttons and ten keys. 
[0031 ] Memory drive 23 drives memory medium stor- 
ing the computer program that is required to perform op- 
eration of navigation processing unit 10 and agent 
processing unit 1 1 . The computer program stored in the 
memory medium includes various programs and data. 
The memory medium may be any one capable of storing 
the computer program. For example, magnetic memory 
medium such as a floppy disk, hard disk and magnetic 
tape, semi-conductor memory medium such as a mem- 
ory tip and IC card, magneto-optteal memory medium 
such as CD-ROM, MO and PD, etc. may be examples 
of suitable memory medium. Any printed matter describ- 
ing the computer program may also used in cooperation 
with a character analyzer. 

[0032] When a rewritable memory medium such as a 
floppy disk and 10 card is used, it may also be used to 
store data in RAM of navigation processing unit 10 and 
agent processing unit 1 1 or memory 29, 30. For example, 
study data (study item data and response data) regarding 
activities of the agent is stored in an 10 card. The driver 
may use the 10 card storing the driver's own data in com- 
mon for two or more vehicles, as far as these vehicles 
have the same device. This means that the agent is pe- 
culiar not to the vehicle but to the driver. 
[0033] Communication control device 24 transmits 
and receives data to and from a portable telephone and 
pemiits communication between the driver and his or her 
friend outside of the vehicle. Communication control de- 
vice 24 also receives data regarding road information 
including traffic jams and traffic accident information that 
are supplied from a traffic infonnation center. It also re- 
ceives Karaoke data. The study data regarding activities 
of the agent may be received and transmitted through 
communication control device 24. 



[0034] Voice output device 25 comprises a plurality of 
speakers arranged in the vehicle through which a voice 
controlled by voice control unit 14 is outputted. The out- 
putted voice includes a voice message for guidance of 

5 the drive route and the agerif s voice or sound that is a 
kind of the agenf s activities. The speakers for audio In- 
struments may be used as voice output device 25. Voice 
control unit 14 changes accent and tone quality of the 
voice outputted by voice output device 25 in response to 

10 the driver's tuning command. 

[0035] A microphone 26 functions as votee input de- 
vice for inputting any voice which is then processed by 
voice control unit 14. For example, a voice for inputting 
the goal of the drive route In navigation processing op- 

15 eration and the driver's voice in response to the agent's 
activities are inputted through microphone 26. A micro- 
phone for Karaoke may be used as microphone 26. How- 
ever, microphone 26 should preferably have directional- 
ity enough to collect the driver's voice. A hand-free unit 

20 comprising speaker 25 and microphone 26 may be used 
for telecommunication. 

[0036] Display 27 is used to represent the results of 
operation by navigation processing unit 1 0, for example, 
guidance to operation, operation menu, operation keys, 

25 a recommended drive route to the goal detemriined by 
the user's input through input device 22, and a guide map 
along the drive route. Display 27 also represents the re- 
sults of operation by agent processing unit 1 1 , for exam- 
ple, the activities of the agent. Any image taken by cam- 

30 era 28 is also shown on display 27 after being processed 
by image processing unit 13. As display 27 may be used 
a CRT display, a liquid crystal display, a plasma display 
or a hologram device that projects a hologram onto a 
front glass. 

35 [0037] Camera 28 comprises CCD cameras that take 
photographs inside and outside of the vehicle. An inside 
CCD camera mounted in the vehicle take photographs 
of the driver. COD cameras are also mounted at the front, 
rear, right side and left side for taking photographs of 

40 surroundings of the vehicle. The photographs taken by 
camera 28 are supplied to image processing unit 13 for 
image analyzing operation. The result of the image an- 
alyzing operation by image processing unit 13 is used to 
determine a program number by agent processing unit 

45 11. 

[0038] Agent data memory 29 stores data and program 
necessary for agent processing operation in accordance 
with this embodiment of the present invention. Memory 
29 comprises memory medium such as a floppy disk, 

50 hard disk, CD-ROM, magneto-optical disk, magnetic 
tape, 10 card, magneto-optical card, etc. and a drive for 
the memory medium used. In a preferred embodiment, 
study item data 292 and response data 293 are stored 
in a portable memory medium such as 10 card and floppy 

55 disk, and other data is stored in a hard disk. 

[0039] Agent data memory 29 stores agent program 
290, program select table 291 , study item data 292, re- 
sponse data 293 and image data 294. Study item data 



5 



9 



EP 0 893 308 B1 



10 



292 and response data 293 represents the result of study 
of the agent through the driver's operation and/or re- 
sponse. Accordingly, study item data 292 and response 
data 293 are peculiar to each driver and renewed when 
the driver makes some operation and/or response. 
[0040] Image data 294 stores the agent appearing on 
display 27. As described before, the agent represented 
on display 27 may be like a human body, like an animal 
such as a chick, dog, cat, frog, rat, etc., robot or any 
imaginary character. The agent may be a child at the 
beginning, which is grown with time as a result of study. 
Image data 294 stores a plurality of the agent appear- 
ance, one of which may be selected by the driver through 
Input device 22. 

[0041] Agent program 290 stores agent processing 
program and communication program in numerical order. 
Agent processing program commands activities of the 
agent. Communication program commands communica- 
tion between the agent and the driver which Is outputted 
through display 27 and speaker 25. 
[0042] Agent program 290 stores plural kinds of voice 
pattems, for example, man's voice, woman's voice, 
child's voice, mechanical sound, animal vorce, famous 
actor's or actress's voice, and popular character's voice. 
The driver can select favorite one of the voice pattems 
stored in agent program 290 through input device 22. 
[0043] Program select table 291 is a table for use in 
selection of a specific one from communication programs 
stored In agent program 290. An example of program 
select table 291 is shown in Fig. 2. Once a specific pro- 
gram is selected in reference to program select table 291 
in Fig. 2, the same program number is applied to an ac- 
tivity pattern select table, an example of which is shown 
in Fig. 3, to detennine the activity (appearance and voice) 
of the agent. The program number of tables in Figs. 2 
and 3 are identical to the communication program 
number stored in agent program 290. 
[0044] Figs. 4A-4D show an example of the agent ac- 
tivities, which is shown on display 27 when one of the 
program Nos, 00001 -00006 is selected (see Fig. 3). This 
activity start with the Image of Fig. 4A which is changed 
to the images of Figs. 4B and 4C, then to the image of 
Fig. 4D that Is the same as the starting image of Fig. 4A, 
This activity is a respectful and ceremonious bow, be- 
cause this is almost the first time that Lisa meets the 
driver. The name "Lisa" is only an example and any name 
may be Inputted by the driver. With the agent's action of 
obeisance appearing on display 27, the agents says 
something that the driver can hear through speaker 25. 
In the program No, 00001 or 00002. the agent makes an 
obeisance while saying "Nice to meet you, I'm Lisa. I wish 
to go with you", which is, however, changed depending 
upon the vehicle's and driver's cun'ent status and as a 
result of the past record study. 
[0045] In this embodiment, input device is also used 
to input the user-related information 298 to be stored in 
agent data memory 29a. The user-related information 
298 may include the user's age, sex, hobby, character, 



favorite regarding the agent, etc. The user-related infor- 
mation 298 stored in agent data memory 29a is read out 
by agent processing unit 1 1 to detemiine the personal 
characteristic of the agent. The user-related infomiation 
298 Is also used to detennine utterance pattem of the 
agent to be outputted through voice output device or 
speaker 25. The user-related information 298 may be 
obtained from the user's response to the agent's ques- 
tion, which is inputted through microphone 25, 
[0046] When current status sensor 40 detects that the 
coolant temperature is lower than a predetemnined level, 
the agent shows a sleepy activity. For example, the agent 
gives or stifle a yawn before making a predetermined 
action (e.g., bow). In another example, the agent shows 
a slower movement and speaking. Such sleepy expres- 
sion may be changed as a result of the study. A cheerful 
activity may be expressed by the agent who speaks loud- 
ly, appears on display while running, for example. 
[0047] The respective items shown on the abscissa of 
table 291 of Fig. 2 detenmlne the program number, on 
the ordinate, to be selected. The items on the abscissa 
may be classified Into two groups, one being determined 
from the vehicle or driver's condition detected by current 
status sensor 40 which includes time, vehicle location 
where ignited, coolant temperature, etc. and the other 
being determined from the study results stored In study 
item data 292 and response data 293 which includes the 
number of today's ignition-on operation, time passage 
from the last ignition-off operation, the total number of 
ignition, etc. In table 291 of Fig. 2, a circle indicates the 
item that is indispensable to select the program of a cor- 
responding number. A hyphened item is not considered 
in selecting the program of a corresponding number. 
[0048] Although Figs. 2 and 3 describes the activity 
pattem (bow) to be represented in response to the ignit- 
ing operation, various activity pattems are likewise de- 
fined in response to some operation. For example, push 
of a brake pedal to an extent greater than a predeter- 
mined level is regarded as a key operation, in response 
to which the agent may fall on his or her behind, take 
several steps backward, keep his or her foot, cries in 
surprise, etc. These responsive activities may be 
changed by the agent's study of sudden braking opera- 
tion. For example, the program detemrilnes that the agent 
falls on his or her behind at the first to third times of sud- 
den braking operation, takes several steps backward at 
the fourth to tenth times, and keeps his or her foot there- 
after. Such program represents that the agent becomes 
experienced and accustomed to the driver's sudden 
braking operation. It may be that when a time of more 
than one week has been passed since the last time sud- 
den braking operation, the agent's activity of one eariier 
stage is applied. For example, where the fifth time sudden 
braking operation occurs ten days after the last time (the 
fourth time) operation, the program detemiines such that 
the agent falls on his or her behind. 
[0049] Study item data 292 and response 293 are 
stored and renewed by the agent's study, examples of 
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which are shown in Figs. 5 and 6, respectively. Study 
Item data 292 shown In Fig. 5 includes the total number 
of Ignition-on, the last ignition-off , today's ignition-on, the 
fuel remainder record, the skip record, the default value, 
etc. 

[0050] The total number of ignition-on operation is in- 
creased by one each time the ignition switch is tumed 
on to ignite the engine. Each time the ignition switch is 
tumed off, its date and time is renewed at the last ignition- 
off data. Each time the ignition-on operation occurs, its 
number (In a day) and its ignition-off time are recorded 
in today's Ignition-on data. When a day is over (at 24: 
00), this data is cleared to zero, awaiting the next Ignition- 
on operation. The time for renewing the today's ignition- 
on operation data is normally 24:00, but may be changed 
upon the driver's demand. 

[0051] The fuel remainder record stores fuel remain- 
ders at the last five times fill-up. Each time the fuel is 
refilled, the newest data is stored at the rightmost column, 
and the last four data are moved to the adjacent left col- 
umns, respectively. The oldest data in the leftmost col- 
umn Is deleted. From the last five fuel remainders may 
be obtained the average fuel remainder (G2). When the 
fuel sensor 415 detects the current fuel remainder (G1) 
is below the average fuel remainder (G2), the agent ap- 
pears on display 27 to show a flll-up suggesting activity 
and says 'IVe got hungry. I want fuel.* through speaker 
25. 

[0052] The skip record data is counted up each time 
the driver refuses or neglects the agent's recommenda- 
tion or communication even when such program number 
has been selected under control of the present Invention. 
The study item data 292 includes the skip record where 
the agent activities of the program number Involves the 
driver's response, such as by Program No. 001 23 where- 
in the agent suggests stop of operation of the air-condi- 
tioner. 

[0053] The default value includes initial values such 
as time, number, temperature, vehicle speed, data, etc. 
The default value is used to initialize data when data has 
been changed by the driver, for example, when the driver 
has changed the Ignition-on operation time from the Initial 
value of 24:00 to his or her desired time, as described 
before. 

[0054] The other data that may be included in the study 
item data 292 include, for example, birthdays of the driver 
and his or her relatives and friends (that can be Inputted 
by the driver), national holidays and their histories, date 
of events such as Christmas Eve. St. Valentine Day, etc. 
Among the communication programs, there may be spe- 
cial menu for special day. In accordance with an example 
of the special menu communication program, the agent 
is appearing on display 27 while dressed as Santa Claus 
on Christmas Eve and Day. 

[0055] The response data 293 in Fig. 6 stores the us- 
er's response history to the agent's activities for the com- 
munication programs which include the user's response 
as one of the study data. The response data have differ- 



ent contents for different communication programs. In an 
example shown In Fig. 6A, the response data for the pro- 
grams of Nos. 001 23 and 001 25 store several latest data 
(time and content). The response data of Program No. 
001 24 stores the only one latest response content, which 
is renewed each time the driver makes a response. In 
other examples, the response data may be several latest 
response contents (not including response time), only 
one latest response data of time and content, only one 
or several latest response date (not Including response 
contents, etc. 

[0056] The symbols A, B and C in Fig. 6A indicate the 
response contents. As shown In Fig. 6B, the symbol A 
indicates "Neglected", B "Refused" and C "Accepted". 
The driver's response content may be discriminated by 
analyze of the driver's voice inputted through microphone 
26 or of infonmation inputted through input device 22. 
[0057] Although in this embodiment the driver's re- 
sponse contents are classified into three pattems, "Ne- 
glected" (A), "Refused" (B) and "Accepted" (C), it may 
have more detailed classification. For example, "Reso- 
lutely Refused", "Angrily Refused", "Pleased", etc. may 
be added. 

[0058] Fig. 7 represents data files stored in navigation 
data memory 30 (Fig. 1). As shown in Fig. 7, navigation 
data memory 30 stores communication area data file 301 , 
map data file 302, crossing data file 303, node data file 
304, road data file 305, route-finding data file 306 and 
photo data file 307. Navigation data memory 30 compris- 
es one or combination of storage medium including a 
floppy disk, hard disk, CD-ROM, magneto-optical disk, 
magnetic tape, 10 card, magneto-optical card, etc., and 
drive units therefor. For example, route-finding data file 
46 Is made of rewritable memory such as flush memory 
while other data files are stored in CD-ROM, and drive 
units therefor are respectively used. 
[0059] Communication area data file 301 stores com- 
munication are data per handy phone types. Communi- 
cation data is used for representation, on display 27, of 
communicatable area by the driver's handy phone con- 
nected or non-connected to communication control unit 
24. This data Is also used while finding out the drive route 
to the goal. The respective communication area data is 
assigned its peculiar number for easy access thereto. 
The communicative area is defined by continuity of short 
segments, so that the communicative area data may be 
specified by a plurality of coordinate data representing 
points connecting the segments. In another example, the 
communicative area is divided into plural rectangles, 
each rectangles being defined by two diagonal points 
thereof, In which case the communicative area data com- 
prises a plurality of coordinate data representing diago- 
nal points of the respective rectangles in the communi- 
cative area. 

[0060] Data in communication area data file 301 is 
preferably renewable to conform with change of the com- 
municative area of the handy phone. To meet this re- 
quirement, the handy phone is connected to communi- 
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cation control unit 24 for communication with an Infonna- 
tion center for updating of the contents of communication 
area data file 301 . In another example, communication 
area data file 301 is made of rewritable and updatable 
memory such as a floppy disk and IC card. 
[0061 ] Map data file 302 stores map data to be repre- 
sented on display 27. Map data is stored in a tree con- 
struction. The uppermost layer stores map data of the 
broadest temtory, and lower layer stores more detailed 
map data of smaller territories. The map data of the re- 
spective layers are assigned their peculiar map codes. 
[0062] Crossing data file 303 stores crossing data such 
as crossing numbers specifying the crossing one by one, 
crossing names, crossing coordinates, road specifying 
numbers start or tenninate with the crossings, existence 
of signals at the crossings, etc. 
[0063] Node data file 304 stores node data specifying 
coordinates of respective points on the respective roads. 
Any road section may be defined by a plurality of node 
data therBon between the beginning and end of the sec- 
tion. 

[0064] Road data file 305 stores road numbers spec- 
ifying the road one by one, the crossing numbers at the 
beginning and end of the respective roads, other road 
numbers of the road that begins or temninates with the 
same point, road width, prohibition information (no park- 
ing, no entrance, one-way road, etc.), number of photo 
data, etc. 

[0065] Data stored In crossing data file 303, node data 
file 304 and road data file 305 are particularly used for 
route-finding. 

[0066] Route-finding data file 306 stores a series of 
crossing data and a series of node data which constitute 
the route found in the navigation system. A series of 
crossing data includes data regarding names and num- 
bers of the crossings on the suggested drive route, num- 
bers of the photos showing these crossings, names and 
numbers of the crossings at which the vehicle should turn 
to follow the suggested drive route, distances between 
the crossings, etc. A series of node data includes data 
regarding coordinates of the respective nodes on the 
suggested drive route. 

[0067] Photo data file 307 stores photos showing the 
crossings, conspicuous landscape and buildings, for ex- 
ample, along the suggested drive route. Photos are as- 
signed their peculiar photo numbers and stored in the 
form of digital, analog or negative films. 
[0068] As shown in Fig. 8, current status sensor 40 
includes an ignition sensor 401 , a vehicle speed sensor 
402, an accelerator sensor 403, a brake sensor 404, a 
side brake sensor 405, a shift position sensor 406, a wink- 
er sensor 407, a wiper sensor 408. a light sensor 409, a 
seat-belt sensor 41 0, a door opening sensor 41 1 , a pas- 
senger sensor 412, a room temperature sensor 41 3, an 
open air temperature sensor 41 4, a fuel remainder sen- 
sor 41 5, a coolant temperature sensor 41 6, an ABS sen- 
sor 41 7, an air-condltloner sensor 41 8, a weight sensor 
419, a fore-distance sensor 420, a rear-distance sensor 



421 . a body temperature sensor 422, a pulse rate sensor 
423, a perspiration sensor 424, a brain wave sensor 425, 
an eye-tracer 426, an infrared sensor 427, and other sen- 
sors 428 for detecting, for example, decrease of air pres- 

5 sure of tires, loosening of belts, opening of windows, op- 
eration of horn, oil temperature, oil pressure, etc. As not- 
ed from the above, current status sensor 40 is used to 
detect various vehicle conditions and driver's conditions 
in this embodiment. In modification, only vehicle condl- 

10 tions or only driver's conditions may be detected by cur- 
rent status sensor 40. Sensors are mounted at suitable 
location inside or outside of the vehicle, respectively. 
Usually one sensor is used for one sensing purpose. 
However, it may be possible that some kind of sensor 

15 receives the detection signal from another sensor to carry 
out indirect sensing. For example, an air-pressure sensor 
may indirectly detect a decrease of an air-pressure of a 
tire in response to a change of a signal from a wheel 
velocity sensor. 

20 [0069] Ignition sensor 401 detects ON and OFF of an 
ignition starter. Vehicle speed sensor 402 may be of any 
conventional type including one wherein a rotation angle 
velocity or revolution of a speed meter cable is detected 
to calculate a vehicle speed. Accelerator sensor 403 de- 

25 tects a degree of push stroke of an accelerator pedal. A 
brake sensor 404 detects a push stroke, push force or 
push speed of a brake pedal to discriminate if the driver 
makes an emergency brake. Side brake sensor 405 de- 
tects operation or non-operation of a side brake. Shift 

30 position sensor 406 detects the current position of a 
transmission shift lever. Winker sensor 407 detects 
■ which winker is lightened. Wiper sensor 408 detects wip- 
er drive conditions (especially, wiper speed). Light sen- 
sor 409 detects various lights and lamps such as head 

35 lights, tail lamps, fog lamps, room lamps, etc. being light- 
ened. Seat-belt sensor 41 0 detects if the driver and pas- 
sengers are equipped with seat-belts. If at least one of 
them is not equipped with a seat-belt, the corresponding 
communication program is working such that the agent 

40 appears on display 27 to give waming, notice, advice, 
etc., one of which is selected by the results of study. 
[0070] Door opening sensor 41 1 detects opening of 
vehicle doors. When detecting a half opening of the door. 
It Is Informed to the driver or passenger by the agenf s 

45 activities (action and voice). Door opening sensor 411 
may comprise plural sensors each detecting opening of 
doors at the driver's seat, at the assistant driver's seat, 
at opposite rear seats, etc., respectively. Passenger sen- 
sor41 2 detects If there are any passengers In the vehicle. 

50 by the photo showing the interior of the vehicle taken by 
camera 28 or by pressure sensor or weight sensor 
mounted under the seat. Room temperature sensor 41 3 
detects a room temperature of the vehicle, and open air 
temperature sensor 414 detects a temperature outside 

55 of the vehicle. 

[0071 ] Fuel remainder sensor 41 5 detects a remainder 
in a fuel tank. As described before, the communication 
program is selected when the fuel remainder is de- 
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creased to the average of the last five records, so that 
the agent urges the driver to refill the fuel. 
[0072] Coolanttemperaturesensor41 Sdetectsacool- 
ant temperature. When detecting that the coolant tem- 
perature is lower than a predetermined level at the time 5 
immediately after the ignition switch Is turned on, the 
agent usually takes a sleepy action. On the other hand, 
when the coolant temperature is higherthan another pre- 
determined level, such communication program Is se- 
lected that the agent looks weary and gives waming or 
notice to the driver to prevent overheating of the engine. 
[0073] ABS sensor 41 7 detects if an ABS (anti-braking 
system) is in operative. As well known, ABS prevents the 
tires from being locked due to emergency braking to 
thereby improve drivablility and stability of the vehicle. 
[0074] Air-conditioner sensor 418 detects the condi- 
tions of an air-conditioner mounted in the vehicle, includ- 
ing ON and OFF thereof, the set temperature and the set 
wind volume, etc. Weight sensor 41 9 detects the driver's 
weight, which is used solely or in combination with the 
photos by camera 28 to specify the driver so that the 
agent* s activities can be detemnlned in accordance with 
study item data 292 and response data 293 Inherent to 
the said driver. When there are different agents selecta- 
ble by plural drivers, one specific agent Is determined by 
identification of the current driver. 
[0075] Fore-distance sensor 420 detects a distance to 
another vehicle or obstacles ahead of the vehicle cun-ent 
position on the road, whereas rear-distance sensor 421 
detects a distance to another vehicle or obstacles astem 
of the vehicle current position on the road. 
[0076] Body temperature sensor 422, pulse rate sen- 
sor 423 and perspiration sensor 424 detects a body tem- 
perature, a pulse rate and perspiration of the driver, re- 
spectively. These sensors may be mounted on the sur- 
face of a steering to detect conditions of the driver's hand. 
Body temperature sensor 422 may be a thermography 
utilizing infrared sensor elements to detect the driver's 
body temperature distribution. Brain wave sensor 425 
detects alpha wave or beta wave of tiie driver to discrim- 
inate the degree of awakening of the driver. Eye-tracer 
traces eye movement of the driver to the driver's intention 
and awakening. Infrared sensor 427 detects movement 
of the user's hands and head. 

[0077] In reference to the flowchart of Fig. 9 showing 
the main routine of tiie agent control operation of the 
device of this embodiment, agent processing unit 1 1 first 
initializes tiie device after Ignition sensor 401 detects ttiat 
tiie ignition switch is turned on. at step S1 1 . Initialization 
includes clearing of RAM, allocation of the respective 
working areas in RAM, loading of program select table 
291 (Fig. 2) into RAM, set of flags to zero, etc. Although 
the operation starts with turning of tiie Ignition switch In 
this embodiment, it may start in response to detection of 
opening or closure of any door by door opening sensor 
411. 

[0078] Next, agent processing unit 1 1 receives the cur- 
rent status data, stores tiiem in tiieir predetemnined areas 



in RAM and analyzes tiiem to confirm the current vehicle 
and user's status, at step SI 2. The cun-ent status data 
include data detected by the various sensors in current 
status sensor 40 and supplied tiierefrom to agent 
processing unit 1 1 tiirough cun-ent status data process- 
ing section 15. The image data obtained by camera 28 
is processed by image data processing section 1 3, which 
is also inputted to agent processing unit 1 1 . The current 
vehicle position data detected by cun-ent vehicle position 
sensor 21 is also inputted to agent processing unit 1 1 
tiirough navigation processing unit 1 0. By way of exam- 
ple of the cun-ent status data, where coolant temperature 
sensor 416 detects a coolant temperature t1. agent 
processing unit 1 1 stores tills temperature t1 In a prede- 
temnined area in RAM and, If the temperature t1 is below 
a predetennined temperature t2, discriminates tiiat tiie 
coolant temperature is low (see the coolant temperature 
column in table of Fig. 2). 

[0079] When ttie driver Inputs any request tiirough mi- 
crophone 26, this is processed by agent processing unit 
11 as a kind of tiie cun-ent status data. Such driver's 
request may be inputted by saying "Phone to Mr. Thin- 
gummy", "Show me restraints around here', "Play a CD', 
etc. The driver's request Is processed by voice process- 
ing unit 14 witii keywords such as "phone", "restraint", 
"play", "CD". These keywonds are an-anged on the ab- 
scissa of program select table 291 in Fig. 2. 
[0080] Agent processing unit 1 1 then checked study 
data (study item data 292 and response data 293) in 
agent data memory 29 to conflmi the agent's study re- 
garding the obtained current status data, at step S13, 
[0081 ] From tiie obtained current status data and tiie 
study data tiiereof, agent processing unit 1 1 discrimi- 
nates if tiiere is any communication program to be initi- 
ated, in reference to program select table 291, at step 
S14. If not (No at SI 4), the procedure is retumed to step 
S12 where it awaits anotfier current status data. Where 
tiiere is a communication program to be initiated (Yes at 
SI 4), it confimis the program number tiiereof. It further 
reads out the driver's response record stored in response 
data 293 for the said program number to thereby deter- 
mine if the communication program of the selected pro- 
gram number should be initiated or suspended, at step 
SI 5. 

[0082] If the answer at SIS is No, agent processing 
unit 1 1 operates to initiate the selected communication 
program so tfiat the predetermined agent's activities are 
shown on display 27 and outputted tiirough speaker 25, 
at step SI 6. This enables good communication between 
the driver and the agent which reflects not only the vehicle 
and driver's current status but also the result of study of 
the past status data and the driver's response records. 
[0083] When the driver shows any response to tiie 
agent's activities made at SI 6. such response data Is 
obtained through microphone 26 or input device 22, at 
step SI 7. 

[0084] Then, agent processing unit 1 1 discriminates if 
the communication program made at SI 6 involves any 
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suggestion, proposal or question that awaits the driver's 
response or answer. For example, agent says "Do you 
want to hear any radio program ?" or "Do you feel 
hungry ? May I find any restraint near here ?". The latter 
program may be initiated in response to detection of a s 
hungry sound through microphone 26. Such a Idnd of 
communication program Is hereinlater referred to as a 
•questioning program'. 

[0085] When the selected communication program is 
not a questioning program (No at SI 8), the procedure is io 
advanced to a step S20. When it is a questioning program 
(Yet at SIS), at a step S19, agent processing unit 11 
controls the agenf s activities in accordance with the driv- 
er's response already obtained at S17. For example, 
when the driver indicates the intention to accept the is 
agenfs proposal "Do you want to hear any radio 
program ?" by answering "Yes", "OK", "Please", etc., 
agent processing unit 1 1 urges the agent to act accord- 
ingly and tums the radio receiver on to follow the driver's 
request. -20 
[0086] Agent processing unit 1 1 accumulates data re- 
garding the execution of the selected communication 
program to obtain the study data, at step 820. For ex- 
ample, when the selected communication program is not 
executed (Yes at S1 5), the number of skip of this program 25 
number stored in study item data 292 is increased by 
one. When the current number of skip is lower by one 
than a predetemiined maximum number, all records now 
stored in study item data 292 and response data 293 for 
that program number should be cleared. Except this 30 
case, the current record in the study item data 292 is 
renewed when the cun-ent status data acquired at Step 
12 includes some study item, and the response data ob- 
tained at Step 1 7 is stored in the response data 293 when 
the iselected communication program is a questioning 35 
program that requests the driver's response. When a pre- 
determined number of response records have already 
been stored in the response data 293, the oldest re- 
sponse data is disused. 

[0087] After executing the above-described steps, at 40 
step S21 , agent processing unit 1 1 confirm that a flag 
has become one in a flag area that was set in RAM at 
Step 11. When the flag is still zero (No at S21), it then 
discriminates if ignition sensor 401 detects that the igni- 
tion switch is tumed off, at step S22). When not detected 
(No at S22), the procedure is retumed to SI 2. When 
tuming-off of the ignition switch is detected (Yes at S22), 
the flag in the flag area in RAM is set to one, at step S23, 
and then the procedure Is retumed to SI 2. When the flag 
is set to one, agent processing unit 1 1 controls such that so 
the agent takes a farewell action that is the final activity 
in the selected communication program (at S16), and 
obtains the study data (at S20). It also execute the au- 
tomatic extinguish any light that the driver has forgotten 
to tum off. Then, the control procedure terminates (Yes ss 
atS21). 

[0088] Figs. 1 0A-IOC show an example of the agent 
activities that is controlled by agent processing unit 1 1 



after the ignition switch Is tumed on. In this example, 
ignition sensor 401 detects that the ignition switch is on, 
current position sensor 21 detects the cun-ent vehicle 
position is other than the driver's home and rest positions. 
These data are supplied to agent processing unit 1 1 as 
the cun-ent status data (at SI 2). Regarding study item 
data 292 and response data 293. it confimis that this is 
the second ignitlon-on operation today, the last agent 
control operation was temiinated at 10:05, on July 12, 
1997, and the total number of ignition-on operation 
amounts to 30 times, at SI 3. 

[0089] From these data, agent processing unit 1 1 de- 
temnines the communication program of No. 00004 
should be executed, in reference to the program select 
table 291 shown in Fig. 2 (at SI 4). The agent activity to 
be outputted through display 27 and voice output device 
25 is an "obeisance" that is a respectful bow, with saying 
"Hello, I'm Usa. I will go with you again". After confirming 
that this communication program should not be suspend- 
ed (No at SI 5). agent processing unit 1 1 starts this com- 
munication program (at SI 6), as shown in Rg. 1 0B. The 
agent's obeisance is represented by a series of stationary 
images or animation. 

[0090] Since the program of No. 00004 is not a ques- 
tioning program that awaits the driver's response (No at 
SI 8), the study data is stored and renewed as a result 
of execution of the program of No. 00004 (at S20). In this 
example, the number of today's Ignition-on operation is 
changed from the previous value '2" to "3", and the total 
number of ignition-on operation is changed from the pre- 
vious value "30" to "31 ". Since the flag Is still zero at this 
time (No at S21) and the ignition is still on (No at S22), 
the procedure is returned to SI 2 so that this communi- 
cation program continues. 

[0091 ] Since the total number of ignition-on operation 
has reached "31 ", the communication programs of Nos. 
00001-00006 can not be selected (see table 291 in Fig. 
2), as far as alt study data (study item data 292 and re- 
sponse data 293) are initialized. Until the total number 
of Ignition-on operation reaches "300". one of the pro- 
grams of Nos. 00007-00019 Is selected in response to 
the cun-ent status data. The agent no more makes an 
"obeisance". The agent makes an ordinary "bow" when 
the coolant temperature is high, while a "sleepy bow" is 
applied when the coolant temperature is low. The agent's 
voice depends on the program number to be selected by 
data such as the place where the ignition switch is tumed 
on. the number of today's ignition-on operation, etc. 
[0092] Figs. 11 A-11 C illustrate another example of the 
agent's activities that may be selected during driving. In 
this example, air-conditioner sensor 41 8 detects that the 
air-conditioner is In operation, and temperature sensors 
413, 414 detects a room temperature T1 and an open 
air temperature T2. respectively. These data are supplied 
to agent processing unit 11 as the current status data. 
Agent processing unit 1 1 1nvestigates the response data 
293 to find that the driver's last twice response to the 
agent's activities of the program No. 00123 are both "C" 
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indicating that the driver accepts the agent's suggestion 
or advice (see Fig. 6A), and reads out the default values 
(T3=2. T4=24) from study Item data 292. From the default 
values T3, T4, it Is confirmed that the temperature rela- 
tionships of T1 -T2 ^ T3 and T2 ^ T4 are satisfied (see s 
Fig. 11 A). 

[0093] Agent processing unit 1 1 thus selects the com- 
munication program of No. 001 23 In reference to the pro- 
gram select table 291 (at SI 4). After confimrilng that this 
program is not a questioning program that awaits the 
driver's response (No at S1 5), it starts executing this pro- 
gram (at SI 6) so that the agenf s activities such as shown 
in Fig. 11 B will be outputted through display 27 and 
speaker 25. More particularly, the agent appears with 
smile on display 27 and says "It becomes cool. May I 
suggest Introducing the open air ?". 
[0094] Agent processing unit 1 1 then acquires the driv- 
ers response to the agents activities (at SI 7). Since the 
program No. 001 23 Is a questioning program that awaits 
the driver's response (Yes at SI 8), it controls operation 
of the air-conditioner to follow the driver's request. When 
the driver's response Is "acceptance", the air-conditioner 
Is tumed off and the front side windows are half-opened 
(at SI 9). 

[0095] After executing control to follow the driver's re- 
sponse, agent processing unit 1 1 stores a result of study 
depending upon the driver's response pattem. Examples 
of the study are shown in Fig. 1 1C. In this case, when 
the driver neglects the agent's suggestion (response pat- 
tem A), application of the program No. 00123 is suspend- 
ed one time. Moreover, the temperature default values 
T3 and T4 to be used in the next application of the pro- 
gram No. 00123 are respectively decreased by one, so 
that the next T3 should be 1 'C and the next T4 should 
be 23'C. When the driver refuses the agent's suggestion 
(response pattem B), application of this program Is sus- 
pended five times, and the temperature default values 
T3 and T4 to be used in the next application of this pro- 
gram are respectively decreased by one, so that the next 
T3 should be 1 'C and the next T4 should be 23*C. When 
the driver accepts the agent's suggestion, the current 
default values T3, T4 remains unchanged, and this pro- 
gram is not suspended but executed when selected. 
[0096] Although in the above description the front side 
windows are half-opened in response to the driver's ac- 
ceptance to the agent's suggestion, It may be possible 
that agent processing unit 11 perfomis more detailed 
control for opening the windows as a result of study. More 
particularly, data regarding window position and degree 
of window opening are also subjected to the agent's 
study. For example, the driver may say "Too much 
opened", "Close a little bit more", "Open rear side win- 
dows too", etc. after half-opening the front side windows 
in response to the driver's acceptance to the agent's sug- 
gestion. Such response reflects the driver's favorite con- 
dition regarding the window opening and, therefore, are 
stored In the response data 293. The default values for 
window opening may be, for example. 1/2 for front side 



windows. When the driver says "Open rear side windows 
too", this is regarded as a kind of the current status data 
(at S 1 2) so that a corresponding communication program 
is executed (at SI 6) to open the rear windows, with the 
con-esponding agent's activities appearing on display 27 
and outputted through speaker 25. 
[0097] In accordance with a modified embodiment of 
the present invention shown in Figs, 12-16, the agent is 
more personified by selectively controlling the agent's 
voice output pattem. In this embodiment, even if the 
same communication program is selected, the agent 
talks to the driver in a different voice pattem or tone, 
depending on the vehicle condition, the user's favorite to 
the agent, the weather condition, etc. The agent* s activity 
or voice pattem may be detemnined by also taking his or 
her character into consideration. 
[0098] In this embodiment, agent data memory 29 in 
Fig. 1 is replaced with another one 29a that stores voice 
data 295, agenf s character select table 296, voice pat- 
tem select table 297 and user-related information 298, 
in addition to agent program 290, program select table 
291 , study item data 292, response data 293 and image 
data 294, as shown in Fig. 12. Except for difference in 
contents of agent data memories 29 and 29a. the device 
is almost similar to that shown in Fig. 1 and therefore no 
description of the other parts and elements of the device 
is given here to avoid tedious repetition. 
[0099] Agent data memory 29a in this embodiment 
comprises one or plural memory medium that can be 
selected from floppy disk, hard disk, CD-ROM, magneto- 
optical disk, magnetic tape, IC card, magneto-optical 
card, etc. and drive unit for the memory medium used. It 
may be preferable that study item data 292, response 
data 293 and user-related data 298 are stored In remov- 
able memory medium such as floppy disk and IC card 
and other data are stored in a hard disk. 
[01 00] Voice data 295 is to output the agent's commu- 
nication through speaker 25. A plurality of predetermined 
words and clauses stored In the voice data 295 may be 
combined upon demand to synthesize the agent's com- 
munication. The driver or user may select appearance 
and voice of the agent through input device 22. The 
agent's voice may be selected, for example, from man's 
voice, woman's voice, child's voice, mechanical voice, 
animal voice, famous actor's or actress's voice, etc. Once 
selected appearance and voice of the agent may be 
changed at any time the user so wishes. When some 
voice pattem is selected. It can be outputted through 
speaker 25 after being tuned by voice control unit 14 In 
accordance with the selected voice pattem. 
[0101] Agent's character select table 296 is used to 
detemiine the agent's character in accordance with the 
user-related Infomiatlon 298. The agent's character se- 
lected in reference to the table 296 Is used as data for 
determining the voice pattem. 

[01 02] An example of the agent's character select ta- 
ble 296 is shown in Fig. 13. As shown in the rightmost 
column of the table 296 in Fig. 13, the agent's character 
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is determined as one selected from the group consisting 
of "obliging", "moderate", "crybaby", "strong sense of jus- 
tice", "beautiful secretary", "competent manager", "high- 
school girl", etc. When a 'beautiful secretary" agent is 
selected, her speaking tends to be placid. When a "high- 5 
school girl" is selected, she usually speaks cheerfully but 
soon become displeased when the vehicle is in a traffic 
jam or exposed to the weather. Of course, these are only 
examples. 

[01 03] The agent's character is selected based on the 
vehicle condition such as the vehicle type and the number 
of Ignrtion-on operation, the user's own data such as the 
user's age, sex, hobby and character, the cun-ent weath- 
er condition, the cun^nt time, etc. These items may be 
inputted by the user through input device 22 and/or ac- 
quired from the user's response (Inputted through micro- 
phone 26) to the agent's question (outputted through 
speaker 25). Some of the items may be automatically 
stored In the table 296 by the driver's vehicle operation 
and/or signals from sensors. Among these items thus 
obtained, the user's own data is also stored in a memory 
section of the user-related infomnation 298 in agent data 
memory 29a. 

[0104] The voice pattern select table 297 is to deter- 
mine the voice output pattern for tuning of the agent's 
voice. An example of the voice pattern select table is 
shown In Fig. 14. As shown in the rightmost column of 
the table 297 in Fig. 14, the agent's voice pattern is de- 
tenntned as one selected from the group consisting of 
"loud", "placid", "cool", "respectful", "tender", "kind", 
"trembling", "angry", "reluctant", "cheerful", "out of hu- 
mor", etc. The voice pattern is selected, in reference to 
the table of Fig. 1 4, from the type of the agenf s commu- 
nication, the vehicle condition, the agenf s character, the 
user's favorite, the extemal factors, the cunrent time, etc. 
[0105] In this example, the agent's communication 
type on the abscissa of the table 297 in Rg. 1 4 is classified 
into at least A-F types, A means route-guidance, B wam- 
ing message. C explanation, D greetings, E call-back, 
and F conversation. The agent's communication type de- 
pends on a specific communication program to be se- 
lected in reference to the program select table 291 of Fig. 
2. In other words, each communication program has its 
data regarding the communication type, that is one of the 
above-described types A-F. 

[0106] The vehicle condition on the abscissa of the 
table 297 In Fig. 14 includes whether or not the vehicle 
Is In a high-speed (say, more than BOkm/hour) drive, 
whether or not the vehicle is out of the prescheduled 
route, whether or not the vehicle interior is noisy (than a 
predetemiined noise level), the total number of Ignition- 
on operation, etc. 

[0107] The agent's character on the abscissa of the 
table 297 in Fig. 1 4 is detemiined in reference to the table 
of Fig. 13, in such a manner as described above. 
[01 08] The favorite on the abscissa of the table 297 in 
Fig. 14 Is selected by the user's input through input device 
22 or inputted from the user's response to the agent's 



question. Tlie driver's favorite may be classified Into 
"young lady", "placid lady", "showy lady", "cheerful girt", 
etc. 

[01 09] The extemal factors also on the abscissa of the 
table 297 in Fig. 14 include the current weather condition 
(fine, cloudy, rain, snow, etc.), whether or not the vehicle 
is in a traffic jam, whether or not the vehicle is driving on 
an uphill or downhill, etc. These extemal factors may be 
automatically inputted by detection of cunrent status sen- 
sor 40. 

[0110] The cun-ent time also on the abscissa of the 
table 297 in Fig, 14 indicates the cunrent time is morning 
(e.g., from the sunrise to noon), daytime (e.g., from noon 
to the sunset), evening (e.g., from the sunset to 23:00) 
or midnight (e.g.. from 23:00 to the next sunrise). The 
cun'ent time classification may be detemnined from de- 
tection by a timer equipped in agent processing unit 1 1 
and a calender (that also stores each day's anticipated 
sunrise and sunset time) stored in agent data memory 29. 
[0111] When the voice output pattem is detennined in 
reference to the table 297, the agent's communication 
(speaking) is tuned accordingly by voice control unit 14. 
How to tune the agent's speaking in accordance with the 
detemnined voice output pattem is not limitative. For ex- 
ample, when the voice output pattem is detennined as 
the loudly-speaking pattem, the agent's communication 
is outputted through speaker 25 with an increased vol- 
ume. In case of the placidly- or respectfully-speaking pat- 
tem, voice control unit 14 controls such that the agent 
speaks slowly so that the driver clearly understands the 
agent's communication word by word. The respectfully- 
speaking pattem may be tuned for giving a silent interval 
at each punctuation mark a little bit longer than by the 
placidly-speaking pattem. In case of the cheerfully- 
speaking pattem, the agent's communication is tuned 
such that the agent speaks in a cheerful and higher voice. 
The agent speaks in a manner that the ending of each 
word is pronounced in a lower voice. 
[01 1 2] The user-related information 298 in agent data 
memory 29a stores the vehicle type of the user* posses- 
sion, the user's age, sex, character, favorite to the agent, 
etc. These user-related data is used to detemnine not 
only the agent's character in reference to the table 296 
of Fig. 13 but also the voice output pattem in reference 
to the table 297 of Fig. 1 4. The user-related data is stored 
for each driver and subject to the study. 
[01 1 3] Fig. 1 5 is a flowchart showing a main routine of 
the agent control operation in accordance with this em- 
bodiment. Agent processing unit 11 first initializes the 
device after ignition sensor 401 detects that the ignition 
switch is tumed on, at step S1 11. This step is almost 
identical to S1 1 of the flowchart of Fig. 9. description of 
which is incorporated herein. 

[01 1 4] Agent processing unit 1 1 then identifies the cur- 
rent driver, at step S1 1 2. More particularly, when the driv- 
er's voice is inputted through microphone 26, it is ana- 
lyzed by voice control unit 1 4 to identify who is the current 
driver. The current driver may be identified by photo taken 
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by camera 28 and then analyzed by image processing 
unit 19. The weight detected by weight sensor 419, the 
driver's seat position, the angle of the room mirror may 
also be taken into consideration to identify the current 
driver. In a preferred embodiment, S1 2 includes a driver- 
Identifying communication program (not shown) in which 
the agent appears on display 27 and speaks through 
speaker 25 for confimiing that the cunrent driver is Just 
as identified. 

[0115] Once the current driver is identified and con- 
firmed, agent processing unit 11 obtains and confirms 
the current status, at step S1 1 3. This step is almost iden- 
tical to SI 2 of the flowchart of Fig. 9, description of which 
is incorporated herein. 

[0116] Then, at step S1 14, agent processing unit 1 1 
perfomns the agent control operation depending on the 
cunrent status conflmied at 8113, in a manner to be de- 
scribed hereiniater in reference to the flowchart of Fig. 
1 6. As a result of the operation at 81 1 4, the agent makes 
some judgement, communication (action and speaking), 
control, study, inspection, etc. It may be possible that the 
agent do nothing when so controlled by the current sta- 
tus. 

[0117] Then, agent processing unit 1 1 discriminates if 
the main control routine should be tenninated, at step 
S1 15. If the main control routine should continue, the 
procedure Is retumed to S113 to repeat the operation 
through 8113-8115. On the contrary, when Ignition sen- 
sor 401 detects that the ignition switch is tumed off (at 
S1 13). this means that the main control routine should 
be tenninated (No at 81 15). In this case, after predeter- 
mined temilnative operation such as tuming-off of room 
interior lamps is made at 81 14, this routine of Fig. 15 is 
terminated. 

[0118] Fig. 16 is a flowchart showing the agent control 
operation that is carried out in 81 14 of the main routine 
of Fig. 15. At step 8121, agent processing unit 11 dis- 
criminates if there is any communication program to be 
initiated, in reference to program select table 291. This 
step is almost identical to SI 4 of the flowchart of Fig. 9, 
description of which is incorporated herein. If there is no 
communication program to be applied (No at 8121), this 
sub-routine is tenninated and retumed to the main routine 
of Fig. 15. 

[0119] When there Is some communication program 
to be initiated (Yes at 8121), it determines the agent's 
character by applying the current status (the number of 
Ignition-on operation, current weather condition, time, 
etc.) that has been obtained at 81 13 of the main routine 
of Fig. 15 and the user-related information 298 in agent 
data memory 295 to the character select table 295 (Fig. 
13), at step 8122. 

[0120] It further determines, at step 8123, the voice 
output pattern of the agent's speaking from the commu- 
nication type of the program that has been confimied at 
S121 ; the agent's character that has been detennined at 
81 22 and the current status (vehicle status, external fac- 
tors, time, etc.), in reference to the voice pattem select 



table 297 (Fig. 14). 

[0121] Thus, at step SI 24, agent processing unit 11 
reads out. from the voice data memory 295, voice data 
necessary to synthesize the agent's speaking contents 

5 (see Fig. 3) of the selected communication program and 
tunes the synthesized speaking contents in accordance 
with the determined voice output pattem. Suppose that 
the program No. 00011 was selected at 8121, in which 
the agent says 'Good aftemoon, I'm Lisa. May I go with 

10 you ? ' (see Fig. 3). And, when the "respectfully-speak- 
ing" voice output pattem was selected at 8123, agent 
processing unit 1 1 controls such that the agent (Lisa) 
says the above greetings relatively slowly. When the 
"cheerfully-speaking" voice output pattem was selected, 

15 the agent says the above greetings in a cheerful and a 
little bit higher tone. 

[0122] After such tuning control operation at SI 24, the 
communication program selected at 8121 is now initiat- 
ed, at step 8125. The prescribed agent's action of se- 
20 lected communication program is shown on display 27, 
while the prescribed agent' speaking thereof Is outputted 
through speaker 25 after subjected to proper tuning at 
8124. 

[01 23] in accordance with this embodiment, the agent 

25 is more personified to make human communication with 
the driver. Even when the same communication program 
is applied, the agent's speaking pattem changes in re- 
sponse to the current status, for example. When the ve- 
hicle is driving with the windows opened, the agent 

30 speaks loudly. When the vehicle is in a traffic jam on a 
rainy day, the agent is out of humor, whereas the agent 
speaks cheerfully during driving at high speed along a 
seaside road (that can be detected by navigation 
processing unit 10) on a fine day. 

35 [01 24] In accordance with another embodiment of the 
present invention, the agent is personified by selectively 
controlling the agent's character depending upon the ve- 
hicle condition or other situation concemed. In this em- 
bodiment, even if the same communication program is 

40 selected, the agent's activity should be changed in re- 
sponse to change of situation. Moreover, the agent's ac- 
tivity is detennined not only the situation but also the past 
record study. 

[0125] In this embodiment, processing unit 1 (Fig. 1) 
45 is connected to agent data memory 29b that stores agent 
program 290, program select table 291 b, study item data 
292, response data 293, image data 294, agent's char- 
acter select table 296 and character-determining data 
299, as shown in Fig. 1 7. Program 290 and data 292-294 
50 are substantially identical to those stored in agent data 
memory 29 of the first embodiment of Fig. 1 . Agent's 
character select table 296 Is substantially identical to that 
stored in agent data memory 29a of the second embod- 
iment of Fig, 12. Accordingly, explanation thereof Is not 
55 repeated herein. Character-detennining data 299 is ba- 
sic data for use in determining the agent's character in 
reference to character select table 296. Such basic data 
Includes the vehicle type, the user's birthday, sex, char- 
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acter for every driver who may use the present system. 
[01 26] Except for difference in contents of agent data 
memories 29 and 29a, the device is almost similar to that 
shown in Fig. 1 and therefore no description of the other 
parts and elements of the device is given here to avoid s 
tedious repetition. 

[0127] Agent data memory 29b in this embodiment 
comprises one or plural memory medium that can be 
selected from floppy disk, hard disk, CD-ROM, magneto- 
optical disk, magnetic tape, IC card, magneto-optical 
card, etc. and drive unit for the memory medium used. It 
may be preferable that study item data 292, response 
data 293 and character-detenmining data 299 are stored 
in removable memory medium such as floppy disk and 
IC card and other data are stored in a hard disk. 
[0128] Program select table 291 b is similar to table 
291 of Fig. 2, but further includes the agent's character 
on the abscissa for some communication programs. This 
means that, in this embodiment, the agenf s character 
that is detenmlned in reference to character select table 
296 comprises one of factors for detenmining a commu- 
nication program to be applied. 
[0129] In this embodiment, as described above, the 
agent activity of a specific communication program that 
is selected by table 291 may be represented in different 
manner depending on the agent's character determined 
by table 296. Suppose that a communication program 
designating the agent's activity of "falling down" Is se- 
lected in response to the driver's emergency braking op- 
eration. Where the agent's character has been deter- 
mined as like a "crybaby", the agent falls down behind 
her buttocks while snivelling or crying. A "beautiful sec- 
retary" do the substantially falls down but remains calm 
and says "Why?". An agent like a "high-school giri" 
screams in doing the action of "falling down". 
[0130] The flowchart of Fig. 15 also describes the main 
routine of the agent control operation of this embodiment. 
Description regarding the flowchart of Fig. 15 should be 
refened to In understanding the agent control operation 
of this embodiment. In this embodiment, however, the 
agent control operation that is carried out at S1 14 of the 
main routine of Fig. 15 is processed as shown by the 
flowchart of Fig. 1 8. 

[0131] Now in reference to the flowchart of Fig. 18, 
agent processing unit 1 1 detennines the agent's charac- 
ter by applying the already obtained current status (e.g., 
the total number of ignition-on operation, the current 
weather, time, etc.) and basic data in character-deter- 
mining data memory 299 to character select table 296, 
at step S221 . 

[0132] From the obtained current status data and the 
agent's character detemriined at S221 , agent processing 
unit 1 1 discriminates if there is any communication pro- 
gram to be initiated, in reference to program select table 
291 b, at step S222. If not (No at S222). the procedure 
is returned to the main routine of Rg. 15. Where there is 
a communication program to be Initiated (Yes at S222), 
it confirms the program number thereof. It further reads 



out the driver's response record stored in response data 
293 for the said program number to thereby determine if 
the communication program of the selected program 
number should be initiated or suspended, at step S223. 
[0133] if the answer at S223 is No, agent processing 
unit 11 operates to initiate the selected communication 
program so that the predetermined agent's activities are 
shown on display 27 and outputted through speaker 25, 
at step S224. This enables good communication be- 
tween the driver and the agent which reflects not only 
the vehicle and driver's cun-ent status but also the result 
of study of the past status data and the driver's response 
records. When the driver makes any response to the 
agent's activities made at S224, such response data is 
obtained through microphone 26 or input device 22, at 
step S225. 

[0134] Then, at S226, agent processing unit 11 dis- 
criminates if the communication program made at S224 
involves any suggestion, proposal or question that awaits 
the driver's response or answer. For example, agent says 
"Do you want to hear any radio program ?" or "Do you 
feel hungry? May 1 find any restraint near here ?'. The 
latter program may be initiated in response to detection 
of a hungry sound through microphone 26. 
[0135] When the selected communication program is 
not a questioning program (No at S226), the procedure 
is advanced to step S228. When it is a questioning pro- 
gram (Yes at S226), at a step S227, agent processing 
unit 1 1 controls the agenf s activities In accordance with 
the driver's response already obtained at S227. Regard- 
ing the example of operation at S227, the description of 
SI 9 should be referred to. 

[01 36] Agent processing unit 1 1 accumulates data re- 
garding the execution of the selected communication 
program to obtain the study data, at step 8228. In con- 
nection with S228, the description of S20 should be re- 
ferred to. Further, in this embodiment, when the response 
at S225 includes any data that influences determination 
of the agent's character, it is stored In character-deter- 
mining data section 299 in agent data memory 29. After 
executing the operation at S228, the procedure is re- 
tumed to the main routine of Fig. 15. 
[0137] Figs. 19A-19C show an example of the agent 
activities that is controlled by agent processing unit 11 
after the ignition switch Is tumed on. In this . example, 
ignition sensor 401 detects that the ignition switch is 
tumed on, current position sensor 21 detects the current 
vehicle position is other than the driver's home and rest 
positions. These data are supplied to agent processing 
unit 1 1 as the current status data at S1 13 of the main 
routine of Fig. 15. Regarding study item data 292 and 
response data 293, it has confirmed that this is the sec- 
ond ignition-on operation today, the last agent control 
operation was temninated at 1 0:05. on July 1 2. 1 997, and 
the total number of ignition-on operation amounts to 30 
times. 

[0138] From these data, agent processing unit 1 1 de- 
termines the communication program No. 00004 should 
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be executed for "beautiful secretary" and No. OOOxx for 
'high-school girt", in reference to the program select table 
291b, at S222 of the flowchart of Fig. 18. The agent ac- 
tivity to be outputted through display 27 and voice output 
device 25 is an "obeisance" that is a respectful bow, with 5 
saying "Hello, I'm Lisa. I will go with you again" in the 
communication program No. 0004 and "Hi, I'm Lisa. Let's 
go with me" in the communication program No. OOOxx. 
After confimning that the selected communication pro- 
gram should not be suspended (No at S223). agent io 
processing unit 1 1 starts this communication program (at 
S224), as shown in Fig. 19B. The agent's obeisance is 
represented by a series of stationary images or anima- 
tion. 

[01 39] Since the selected program of No. 00004 or No. is 
OOOxx Is not a questioning program that awaits the driv- 
er's response (No at S226), the procedure is advanced 
to S228 where the study data Is stored and renewed as 
a result of execution of the selected program. In this ex- 
ample, the number of today's ignition-on operation is 20 
changed from the previous value "2" to "3", and the total 
number of Ignition-on operation is changed from the pre- 
vious value "30" to "31 Then, the procedure is returned 
to the main routine of Fig. 15. 

[0140] Fig. 20A represents a example of program se- 25 
lect table 291 b, and Fig. 20B represents the agent* s ac- 
tivities to be outputted in the communication program 
selected by table 291b of Fig. 20A. The communication 
program Nos. 02000-02003 shown in Fig. 20A is selected 
when the driver changes the shift lever from the parking 30 
position (P) to the driving position (D) while the side brake 
is still in operation and then pushes the accelerator pedal 
to a certain degree. The accelerator pedal operation, the 
side brake operation and the shift lever position are de- 
tected by sensors 403, 405 and 406 (Fig. 8) in current 35 
status sensor 40 (Fig. 1 ), which agent processing unit 1 1 
confirms at S113 of the main routine of Fig. 15. In the 
communication program 02000-02003, the agent calls 
the driver's attention to the current side brake position 
and immediate release thereof, which is accomplished 40 
by the agenf s activities shown in Fig. 20B. 
[0141] When the "obliging beautiful secretary" has 
been selected as the agent's character (at S221 of Fig. 
18), the communication program No. 02000 is selected 
In reference to program select table 291 b of Fig. 20A (at ^5 
S222). As shown in Fig. 20B, this communication pro- 
gram designates the agent activity such that the agent 
warns by saying "The side brake is still in operation. Re- 
lease it, please", in a persuasive tone. Also, the agent's 
activity is shown on display 27 just like such that her right so 
hand is pulled back by the side brake. 
[0142] When the "high-school girl' has been selected, 
the communication program No. 02001 is applicable. In 
this case, the high-school girl on display takes the same 
reaction while saying "Oh no! The side brake is still work- ss 
ing. Release It, please" in a cheerful tone. When 
the "crybaby" has been selected, the communication 
program No. 02002 is applicable so that the crybaby 



takes the same reaction with a tearful face and 
says "Ouch! The side brake is stilt in operation. Release 
it, please" In a sadly and entreating tone. When 
the "placid" agent has been selected, the communica- 
tion program No. 02003 is applicable. In this case, the 
agent takes the same reaction and says "May I suggest 
you releasing the side brake" in a placid tone. 
[01 43] In the foregoing embodiment, the agent's char- 
acter changes depending upon the current status, the 
user-related data, the weather condition, time, etc. How- 
ever, In a more simple manner, the agent's character 
may be selected from predetemnined several characters 
depending on the driver's familiarity to the agent system. 
The driver's familiarity to the agent system is calculated 
in a manner to be described hereinlater. The agent is 
"placid" to the driver who Is much familiar to the agent 
system, whereas an "obliging" agent appears to the un- 
familiar driver. The obliging agent provides suggestion, 
advice, help, explanation, etc. in a meddlesome manner 
rather than by the placid agent. 
[01 44] In this modified embodiment, agent data mem- 
ory 29c stores familiarity data 300, as shown in Fig. 21 . 
Agent's character select table 296 and character-deter- 
mining data 299 in agent data memory 29b are not 
present. Familiarity data 300 is stored for every driver 
who may use the agent system, and renewable by the 
past record study. In this modified embodiment, the agent 
character is "obliging" or "placid" so that only these two 
characters are included in the column of "Agent's Char- 
acter" in program select table 291 b (Fig. 20A). 
[0145] Fig. 22 shows an example of familiarity data 
300. As shown, familiarity data 300 describes the pro- 
gram number, the flag, the last five time response period, 
the average response period, the shortest response pe- 
riod and the number of response cancellation for each 
communication program. The response period means a 
period of time from when the agent makes some question 
until when the driver makes some response thereto, or 
a period of time when the driver is expected to make 
some operation until the said operation is actually taken. 
The response periods for the latest five time responses 
are recorded in familiarity data 300. The average re- 
sponse period is the average of the five records. The 
shortest response period is the shortest one of the five 
records. When the driver cancels his or her response or 
operation while the selected communication program is 
still working, it is accumulatively recorded as the number 
of response cancellation. The flag describes how many 
times this communication program has been selected. 
Familiarity data 300 is prepared for every communication 
program that requests the user's response, for every user 
of this agent system. 

[01 46] The familiarity level is calculated from these da- 
ta and also described in familiarity data 300. Of course, 
the familiarity level is zero when the flag is zero. Each 
time the communication program is selected, the famili- 
arity level is increased by 1 point. The average response 
period becomes shorter than a predetermined period, 
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the familiarity level is increased by 2 point. The shortest 
response period becomes lower than a predetermined 
period, it is increased by 1 .5 point. 
[01 47] On the other hand, when the average response 
period becomes longer than a predetermined period, the 
familiarity level Is decreased by 2 point. The shortest re- 
sponse period exceeds a predetermined period, it is de- 
creased by 1.5 point. It is decreased by 1 point, when 
the response cancellation is made by the driver. The re- 
spective points to be increased or decreased may be 
detemnined as desired. 

[01 48] In this modified embodiment, agent processing 
unit 1 1 detemriines the agent* s character by confirming 
the familiarity level of the selected responsive program 
and of the cun-ent user, at S221 of Fig. 18. By way of 
example, when the familiarity level in familiarity data 300 
(Fig. 22) Is below 40. the agenf character Is detemnined 
as "obliging", whereas it is 70 or more the agenf s char- 
acter is detemriined as "placid". In accordance with the 
agent's character thus detemiined at S221, a specific 
communication program is selected atS222 followed by 
the succeeding steps. 

[01 49] At S225, agent processing unit 1 1 confimis the 
driver's response to the agent's activity in the running 
program, which is made through microphone 26 or input 
device 22. It also confirms the response period from when 
the agenfs activity Is outputted through display 27 and 
speaker 25 until when the driver's response thereto is 
obtained. From these, agent processing unit 11 rewrite 
or renew the familiarity data 300. More particularly, in the 
column of the last five time response period, the oldest 
record Is annulled, the remaining four records are moved 
to the older boxes one by one and the obtained response 
period is recorded as the latest one. Then, the average 
response period and the shortest response period are 
calculated and recorded. From the renewed records, the 
familiarity level is calculated by a predetermined formula 
and the newly obtained familiarity level is recorded In the 
familiarity data section 300 which should be used in the 
next detennination of the agent's character. 
[01 50] In accordance with still another embodiment of 
the present Invention, a plurality of agents may appear 
on display 27 and/or outputted through speaker 25 so 
that the user can easily understand the communication, 
message, suggestion, etc. from the agents. 
[0151] In this embodiment, processing unit 1 (Fig. 1) 
is connected to agent data memory 29d that stores agent 
program 290, program select table 291, study Item data 
292, response data 293, image data 294, help-agent pro- 
gram 301, help-agent program table 302 and familiarity 
data 300, as shown In Fig. 23. Program 290. table 291 
and data 292-294 are substantially identical to those 
stored in agent data memory 29 of the first embodiment 
of Fig. 1 . Familiarity data is substantially identical to that 
stored in agent data memory 29c of the preceding em- 
bodiment of Fig. 21 and an example thereof is shown in 
Fig. 22. Accordingly, explanation thereof is not repeated 
herein. 



[01 52] Help-agent program 301 comprises help-agent 
processing program and help-communication program 
especially for each responsive communication program 
stored in agent program section 290. Help-agent 

5 processing program is used to carry out help-agent op- 
eration which provides explanation or other suitable com- 
munication for easier understanding of the main-agent's 
activity, while some responsive communication program 
Is working. Help-agent communication program de- 

10 scribes the help-agent's activities to be represented 
through display 27 and speaker 25. 
[0153] Help-agent program 301 stores plural kinds of 
voice data for each main agent's voice of the main-com- 
munication program, one of which may be selected by 

15 the driver's operation through input device 22, for exam- 
ple, so that the driver distinguishes the main-agent's and 
help-agent's voices both outputted through speaker 25. 
For example, the main-agent is female, the help-agent 
should be male. In a preferred example, after the maln- 

20 agent's voice is determined, agent processing unit 1 rep- 
resents on display 27 a list of voice pattems for the help- 
agent which is distinguishable from the main-agent's 
voice, and the driver select one through input device 22. 
In the absence of the driver's instruction, agent process- 
es ing unit 1 automatically detennines the help-agent's 
voice that is distinguishable from the main-agent's voice. 
Likewise, the help-agent's appearance should be well 
distinguishable from the main-agenf s appearance. 
[0154] Help-agent program table 302 represents cor- 

30 respondence between the (main-) agent program 
number and the help-agent program number, and the 
familiarity level that is critical to carry out the help-agent 
program. An example of the help-agent program table 
302 is shown in Fig. 24, and some of the help-agenf s 

35 activities in the help-agent programs in Rg. 24 are rep- 
resented in Fig. 25. 

[0155] For example, when the agent's communication 
prpgram No. 00123 (that suggests stop of the air-condi- 
tioner) is in operation, the help-agent program No. 5001 0 

40 is applicable provided that the driver's familiarity to this 
communication program is 50 or lower. In this help-agent 
program, the help-agent points his or her forefinger at 
the main agent (Lisa) on display 27 and says "Lisa stops 
the air-conditioner and opens the front windows half" 

45 through speaker 25. Even when the same communica- 
tion program No. 00123 is wori<ing, when the driver's 
familiarity is higher than 50, the con-esponding help- 
agent program No, 50010 is not operative and no help- 
agent appears. 

50 [01 56] Fig. 26 is a flowchart of the main routine of the 
agent control operation in accordance with this embodi- 
ment. Steps S301-306 of this routine are substantially 
Identical to S1 1-1 6 of the flowchart of Fig. 9. so that de- 
scription of the latter should be referred to as description 

55 of the fonner. 

[0157] After initiating the applicable communication 
program at S306, agent processing unit 11 checks and 
confinns the familiarity data 300 of the said communica- 
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tion program in agent data memory 29, at step S307. At 
the same time, the familiarity flag is set to 1 . Then, the 
program number of the selected communication program 
and the confirmed familiarity data are applied to the help- 
agent program table 302 (Fig. 24) to discriminate if there 
is any help-agent program to be Initiated, at S308. If not 
(No at S308), the procedure is advanced to step S310 
where the driver's response is obtained. Steps 310-316 
are substantially identical to S 17-23 of the flowchart of 
Fig. 9 and, therefore, no detailed description is given 
herein. 

[01 58] When some help-agent program to be initiated 
in the table 302 (Yes at S308), this is executed so that 
the prescribed help-agent's activity (action and utterance 
such as shown in Fig. 25) is outputted through display 
27 and speaker 25, at step S309. 
[0159] In accordance with this embodiment, the help- 
agent appears and supports the main-agent's activity 
when the driver Is not so much familiar to the present 
agentsystem, whereby the driver can make more smooth 
communication with the agent and more suitable re- 
sponse to the agent's question or suggestion. 
[0160] Figs. 27A-27C illustrate an example of the 
agent control operation made by the flowchart of Fig. 26. 
This example Is executed when the air-conditioner sen- 
sor 41 8 detects "ON", the room temperature sensor 413 
detects the room temperature T1 and the open air tem- 
perature sensor 414 detects the open air temperature 
T2, as the current status obtained at S302. Operation at 
S303 reveals that the last two response to the commu- 
nication program No. 00123 are both C indicating the 
driver's acceptance so that the default temperature val- 
ues T3=2 and T4=24 have been read out from the study 
item data 292. Consequently, The relationship of (T1 -T2) 
^ T3 and T2 ^ T4 are both satisfied, as shown in Fig. 
27A. 

[01 61 ] From these operation at S302 and S303, agent 
processing unit 1 1 finally selects the communication pro- 
gram No. 001 23 at S304. Then, after confirming that this 
program is not subjected to suspension at S305, it starts 
the communication program No. 00123 for representa- 
tion of the (main-) agent* s activity through display 27 and 
speaker 25, at S306. 

[0162] Next, agent processing unit 11 confimis the 
driver's familiarity to the selected communication pro- 
gram at S307 and finds out an applicable help-agent pro- 
gram at S308. Since the driver's familiarity to the com- 
munication program No. 00123 is 25 (Fig. 27A) in this 
example, and the communication program No. 00123 
has a corresponding help-agent program No. 50010 in 
the help-agent program table 302 (Fig. 24), this help- 
agent program No. 50010 is executed at S309. Accord- 
ingly, the help-agent H appears on display 27 in a pre- 
determined appearance and speaks through speaker 25 
in a predetemiined manner and tone. In this example, 
the main-agent (Lisa) E says "It becomes cool. I may 
suggest introducing the open air." and then the help- 
agent H says "Lisa stops the air-conditioner and opens 



the front windows half". The help-agenfs activity pro- 
vides support and explanation of the main-agent's activ- 
ity to the driver. Data necessary to display the help-agent 
activity is included in the image data 294. 

5 . [0163] At S310 agent processing unit 1 1 obtains the 
driver's response. Since the communication program No. 
00123 is a questioning or responsive program (Yes at 
S31 1 ). it controls such that the air-conditioner Is stopped 
and the front windows are opened half in response to the 

10 driver's acceptance to the agent's suggestion (Fig, 27B), 
atS312. 

[01 64] The driver's response is stored In response da- 
ta 293 for the communication program No. 00123, at 
S31 3. Since response data 293 stores the latest two re- 

15 sponse, the older response data is deleted and the newly 
obtained response data is stored together with one re- 
maining data. As shown in Fig. 27C, when the driver's 
response is "neglect", application of this communication 
program is suspended one time, and the default temper- 

20 ature values T3 and T4 to be next applied are decreased 
by one, respectively. When the driver shows "refusal", 
application of this communication program is suspended 
five times, and the default temperature values T3 and T4 
to be next applied are decreased by one, respectively. 

25 In case of "acceptance", the same default values remain 
unchanged in the next application. 
[01 65] Operation at S31 2 also includes re-calculation 
of the driver's familiarity to this communication program 
No. 00123. More particulariy, the familiarity data 300 (Fig. 

30 22) is rewritten such that the oldest response period Is 
deleted and the newly obtained response period (I.e., a 
period of time from when the agent appears on display 
27 and until any driver's response is obtained). From the 
rewritten five response period data, new average re- 

35 sponse period, new shortest response period and new 
familiarity level are calculated and recorded in the re- 
spective memory areas. 

[01 66] When the driver's familiarity to the communica- 
tion program No. 00123 is grown to above 50, agent 

40 processing unit 1 1 discriminates there is no applicable 
help-agent program at S308, because the help-agent 
program table 302 teaches that the help-agent program 
No. 50010 corresponds to the communication program 
No. 00123 but should be executed only when the driver's 

45 familiarity is lower than 50. In this case, only the main- 
agent (Lisa) appears on display 27 while saying the pre- 
scribed suggestion to introduction of the open air, in ac- 
cordance with the communication program No. 00123. 
Then, agent processing unit 1 1 obtains the driver's re- 

50 sponse at S310. make suitable control in conformity to 
the driver's response at S31 1 and rewrites the response 
data 293 and the familiarity data 300, as in the above- 
described manner. 

[0167] The above-described embodiment may be 
55 modified as follows. In this modification, a plurality of sub- 
agents may appear depending on the driver's trend. A 
typical example of the sub-agent is a short-cut agent who 
appears, when the driver is much familiar to a specific 
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communication program, for short-cut or skip of a part of 
the agent's lengthy activity of the same communication 
program. 

[01 68] In this modification, agent data memory 29d of 
Fig. 23 is replaced with agent data memory 29e of Fig. 
28. In agent data memory 29e are stored agent program 
290. program select table 291 , study item data 292, re- 
sponse data 293, Image data 294, sub-agent program 
303. sub-agent program table 304, trend data 309 and 
frequent program data 305, as shown in Fig. 28. Since 
the fomier five memory contents have been described 
in connection with the preceding embodiments, the last 
two will be described in detail. 
[0169] Sub-agent program 303 comprises sub-agent 
processing program and sub-agent communication pro- 
gram especially for each communication program stored 
in agent program section 290. Sub-agent processing pro- 
gram Is used to carry out sub-agent operation wherein a 
sub-agent appears together with a main agent depending 
on the driver's trend during operation of some commu- 
nication program. Sub-agent communication program 
describes the sub-agenf s activities to be represented 
through display 27 and speaker 25. Sub-agent process- 
ing program at least includes a program for achieving the 
short-cut agent operation that short-cuts or skips the 
main-agent's lengthy activity when the driver is so familiar 
thereto. 

[0170] In another example, a specific trend value is 
obtained from the driver's birthday and the traffic jam 
condition so that the today's fortune is noticed to the driv- 
er when the vehicle is in a heavy traffic jam. 
[01 71 ] In still another example, data regarding the driv- 
er's favorite professional baseball team and the radio 
program broadcasting the said team's games are stored 
as the driver's trend data 309. The sub-agent program 
is executed at the time when the radio program is to start, 
so that the sub-agent announces the radio program and 
the opponent team and ask if the radio receiver should 
be tumed on. The radio program data may be Inputted 
by the driver or automatically received from communica- 
tion control unit 24. 

[01 72] Another driver's trend data may be detemiined 
from the current vehicle position and the places of the 
driver's interest so that the sub-agent draws the driver's 
attention to some recorded place near the current vehicle 
position. Still another driver's trend data may be deter- 
mined from the driver's driving career and the cun-ent 
vehicle position so that the sub-agent announces "Be 
careful, you are driving on a winding road", for example. 
[0173] Most typical example of the sub-agent program 
will be the short-cut agent program to be described in 
more detail. 

[01 74] After executing the sub-agent program at S408, 
agent processing unit acquires the driver's response, at 
S409. As described before, if the driver's trend data is 
lower than the predetermined value (No at S407), no sub- 
agent program is executed and the driver's response Is 
obtained at S409. The following steps S410-S415 are 



substantially identical to S18-S23 of Fig. 9. 
[0175] The activity In the sub-agent program will now 
be described in detail. In the following description, the 
sub-agent is the short-cut agent. As shown in Fig. 33, 

5 the communication program for rnenu selection compris- 
es multi-stage operation. The first-stage operation is 
mode select operation. The second-stage operation in- 
cludes goal select mode and system select mode, one 
of which may be selected by the driver's operation 

10 through input device 22 when the first stage operation Is 
wori<ing. When the goal select mode is selected, then 
the driver may select and enter one of third-stage oper- 
ation, that is. how to select the goal, from Its category 
(highway lamps, railroad stations, hotels, restraints, golf 

15 courses, etc.), from its phone number or from its spelling. 
When the system select mode is selected, then the driver 
may select one of third-stage operation for setting dis- 
play, voice or communication in the agent system. In the 
normal operation, operation begins with at the first stage, 

20 proceeds through the second stage and temninates at 
the third stage. 

[0176] Fig. 34 is similar to Fig. 33 but different in that 
there is a short-cut program shown by dotted lines. In 
this embodiment, after selecting the first-stage operation 

25 for mode selection, the driver may skip over the second- 
stage operation to select one of the third-stage operation 
by using the short-cut program. The short-cut program 
is applicable when the driver is so familiar to the com- 
munication program, or when the driver's familiarity to 

30 the communication program is 75 or more In this exam- 
ple. 

[0177] When the driver so operates to start the navi- 
gation system, agent processing unit 1 1 first perfomis 
the first-stage operation, that is the mode select opera- 

35 tion. The communication program No. 00540 now be- 
comes operative (at S406) so that the main agent ap- 
pears on display 27 and says "May I soon start navigation 
program or provide you system select program Since 
this communication program has no corresponding 

40 short-cut program (No at S407), it awaits the driver's re- 
sponse at S409. When the driver requests navigation 
program In reply to the agent's activity of this program 
No. 00540 through Input device 22 or microphone 26, 
this is confimied at S409. This program No. 00540 is not 

45 subjected to the short-cut control (No at S41 0), and agent 
processing unit 1 1 carries out the following operation at 
S41 1-415, then the procedure is retumed to S402. 
[01 78] The communication program No, 00550 for se- 
lecting the goal input mode Is now executed at S406. 

50 This means that the operation proceeds to the second- 
stage operation of Fig. 33. In this program No. 00550. 
the main agent appears on display 27 and says "Select 
your goal input mode. From its category, from its phone 
number or from Its spelling ?', as shown in Fig. 36. Agent 

55 processing unit 1 1 then compares the driver's familiarity 
(75) with the predetemiined critical value (70 in Fig. 29) 
at S407. Since the former is larger than the latter in this 
example (Yes at S407), in reference to the short-cut pro- 
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gram table 304 in Fig. 29, agent processing unit 1 1 ex- 
ecutes the short-cut operation in accordance with one of 
the short-cut programs selected in accordance with the 
frequent program of the last five program selection (at 
S408). 

[0179] Fig. 35 is a flowchart of the short-cut agent con- 
trol operation at S408. As shown, agent processing unit 
1 1 obtains which is the most frequent program, at S481 
from the frequent program data 305 in agent data mem- 
ory 29. Since the most frequent program is No. 00580 in 
the example of Fig. 31 , which means that the driver nor- 
mally Inputs the goal from its category in response to the 
agent's activity of the mode select program No. 00540. 
agent processing unit obtains the short-cut program No. 
8001 0 in reference to Fig. 30, at S482, and then executed 
the same at S483. By executing the short-cut program 
No. 8001 0, the short-cut agent S appears on display 27 
beside and behind the main agent E, and says 'You can 
readily enter the categorical input mode." following the 
main agent's speaking, as shown in Fig. 36. The short- 
cut agent then awaits the driver's response. 
[0180] When the driver responsively says "Restraints", 
this is confimied by the short-cut agent. In this case, 
agent processing unit 11 compulsory temninates the main 
agent communication program No. 00550 without reev- 
ing any response from the driver, and execute another 
communication program No. 00600 (at S404). The com- 
munication program No. 00660 is prepared as the fourth- 
stage operation after selecting the categorical input 
mode operation 00580 as one of the third-stage operation 
(Figs. 33 and 34). Thereafter, the current vehicle position 
is detected and the restraint map near the detected cur- 
rent vehicle position is shown on display 27 for selective 
input by the driver. 

[0181] The driver may say "Category" in response to 
the main- and sub-agent's activities at S483. in this case, 
the procedure is advanced to the categorical input mode 
operation of the program No. 00580 which is executed 
at S404. The list of selectable categories is represented 
on display 27 and the main agent says "Select category 
of the goal'. When the driver selects a specific input 
mode, the operation proceeds in the above-described 
manner. 

[0182] On the other hand, if the agent's familiarity is 
smaller than 70 (No at 8407), agent processing unit 11 
does not execute the short-cut program. In this case, only 
the main agent appears and says in accordance with the 
main agent communication program that has been de- 
temiined at 8404, as shown in Fig. 37. 
[0183] In this embodiment, the short-cut agent ap- 
pears together with the main agent to suggest the short- 
cut operation to the driver familiar to the agent system. 
When the driver forgets the short-cut operation, he or 
she may follow the main agent's activity to successively 
carry out the multi-stage operation stage by stage. 
[0184] The sub-agent may act independently of the 
main agent's activity. For example, the main agent acts 
as a guide or navigator whereas the sub-agent notices 



the cun^ent vehicle condition to the driver, and the vice 
verso. 

[01 85] In accordance with yet another embodiment of 
the present invention, one or more of personified agent 

5 appears on display 27 together with a predetermined 
background so that the user can easily understand the 
communication, message, suggestion, etc. from the 
agents. The agent's activity and the background are de- 
termined depending on the current status of the vehicle, 

10 the driver and passengers, etc. as well as the driver's 
response. 

[0186] In this embodiment, processing unit 1 (Fig. 1) 
is connected to agent data memory 29f that stores agent 
program 290, program select table 291 , study Item data 

15 292f, response data 293, image data 294, background 
select table 306 and response recognition data 307, as 
shown In Fig. 38. Program 290, table 291 and data 
293-294 are substantially identical to those stored in 
agent data memory 29 of the first embodiment of Fig. 1 . 

20 Accordingly, explanation thereof is not repeated herein. 
[01 87] The study item data 292f in agent data memory 
29f is shown by example in Fig. 5 which is slightly different 
from the study item data 292 (Rg. 5) in agent item data 
292 (Fig. 1). That is, the study item data 292f also in- 

25 eludes audio instruments equipped in the vehicle and 
their operating condition. Audio instruments may com- 
prise a radio receiver, a CD player, a MD player, a cas- 
sette tape player, a TV set, etc. The audio instmment 
operating condition describes when and where any audio 

30 instrument is tumed on. If the radio or TV is turned on, 
the selected channel is also recorded. The audio instru- 
ment operating condition have five time records. The re- 
maining items in the study item data 292f have been de- 
scribed in connection with the first embodiment. 

35 [01 88] In this embodiment. Image data 294 include da- 
ta for representing the background on display 27, as well 
as data for the agenf s appearance and activity. The pho- 
to image taken by camera 28 is analyzed by image 
processing unit 13 to be used to represent the agent's 

40 activity or the background. The background image data 
294 may be photo, animated or any other image data 
representing sunrise, sunset, stariight night, snowy 
mountain, autumnal tints, golf course, plateau, etc. 
Among these, sunrise, sunset and stariight night back- 

45 grounds may represent the current time, snowy mountain 
and autumnal tints may represent the cunent season and 
golf course and plateau background may represent the 
goal inputted by the driver in the navigation system. The 
background may also be the agent's belongings, such 

50 as nameboards showing the driver's selectable response 
or answer "YES" and "NO", which is applicable when 
some questioning communication program is in opera- 
tion. When the driver is listening to the music through CD 
or radio, a background like a musical note may be shown 

55 on display 27. 

[01 89] An example of the background select table 306 
is shown in Fig. 40. The items on the ordinate are various 
background, among which one or more of specific back- 
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ground is selected by the items on the abscissa. The 
Items on the abscissa Include, in this example, the current 
time, the cunrent season, the Instruments now In opera- 
tion, the voice recognition status, the most frequent rec- 
ognized word, the goal, etc. These items may be detected 5 
by various sensors included in status sensor 40, alone 
or In combination with the contents in the study item data 
292 and/or response data 293. In this table, the Item sym- 
bolized by "O" is a necessary condition for selecting the 
background on the ordinate. The item with no symbol io 
should not be satisfied to select the background on the 
ordinate. 

[0190] Figs. 41A-41C show examples of the back- 
ground to be selected by the background select table 
306, which is represented together with the agent's ac- ^5 
tivity on display 27. In Fig. 41 A, the background with a 
red frame (see Rg. 40) Is selected when the driver's voice 
is not recognizable by the agent The agent in Fig. 41 A 
Is sitting down, which represents that the vehicle is now 
at a standstill. The background is shown in Fig. 41 B with 20 
a yellow frame (see Fig. 40) which indicates that the agent 
awaits the driver's answer by saying "YES" or "NO". In 
this background condition, the driver's voice other than 
"YES" and "NO" can not be recognized by the agent. The 
agent in Fig. 41 B carries two nameboards of "YES" and 25 
■NO" while saying "Please answer by 
saying "YES" and "NO* to my question", for example. 
This agent's activity Is outputted when some questioning 
communication program that requestthe driver's answer. 
The agent's belongings such as the above nameboards 30 
should be a kind of the background, as noted from Fig. 40. 
[0191] When the communication program for goal in- 
put is running and the driver selects to input the goal by 
its category and then by its distinct, the agent appears 
on display 27 with a nameboard of "DISTRICT" for urging 35 
the driver to input a specific name of the distinct of the 
goal, as shown in Fig. 41 C. The driver's input may be by 
voice through microphone 26 or by operation of input 
device 22. At the same time, a pertinent map is shown 
on display 27, and the frame of display 27 is colored 40 
green for indicating that any driver's voice is recogniza- 
ble. In this example, the "DISTINCT" nameboard, the 
background map and the green frame are represented 
as the backgrounds that have been selected by the table 

306 (Fig. 40). Fig. 8D merely shows another example of 45 
the background with a green frame indicating voice rec- 
ognizable condition, as above described. The agent is 
standing upright in Figs. 41B-41C which Indicates that 
the vehicle is running. 

[01 92] The response recognition data 307 represents so 
a result of the agent* study of the response from the driver. 
The response recognition data 307 is prepared for each 
driver and rewritten as the driver makes some fresh re- 
sponse. 

[0193] An example of the response recognition data 55 

307 is shown in Fig. 42. As shown, the response recog- 
nition data 370 includes the response recognition results, 
distribution of the recognized response pattems and the 



most recognizable response pattem, respectively for af- 
firmative and negative responses. The affirmative re- 
sponse may be "Yes", "OK", "Well", "Certainly", "Sure", 
etc. and the negative response may be "No", "You're 
Kidding", "Not at all", etc. The response recognition re- . 
suits records in respective answer columns "O" or "X" 
symbols indicating if such driver's answer could be cor- 
rectly recognized in the agent system, for past 10 times 
recognition. The driver's answer inputted through micro- 
phone 26 is processed by voice processing unit 14. and 
a signal after voice processing is supplied to agent 
processing unit 1 1 for executing some control operation. 
If the driver's answer has been recognized in error, the 
driver's should make any surprising response to the ex- 
ecuted control operation. From this, it could be judged 
that the last time voice was recognized in error. 
[0194] The distribution percentage is determined by 
dividing the total number of voice acquisition by the 
number of con-ect voice recognition, followed by being 
multiplied by 100. The driver's answer having the great- 
est distribution percentage is recorded as the most rec- 
ognizable answer. Thus, two nameboards describing the 
most recognizable affinnative and negative answers in 
their response recognition data 307 (Rg. 42) are shown 
as the background of display 27. in case of Fig. 41 B, for 
example. 

[01 95] The main routine of the agent operation of this 
embodiment is the same as shown in Fig. 1 5 and. there- 
fore, description regarding the flowchart of Fig. 1 5 should 
be referred to in understanding the agent control opera- 
tion of this embodiment. In this embodiment, however, 
the agent control operation that is carried out at S1 14 of 
the main routine of Fig. 1 5 is processed as shown by the 
flowchart of Rg. 43. In he flowchart of Fig. 43, steps 
S501-502 are substantially identical to S14-15 of the 
flowchart of Fig. 9, so that description of the latter should 
be referred to as description of the former. 
[0196] After confimiing that the applicable comnriuni- 
cation program should be executed in this operation (No 
at S502), agent processing unit 1 1 discriminates if the 
applicable communication program is an instrument con- 
trol program, at step S503. The instrument control pro- 
gi^am means a program to be executed to control oper- 
ation of the vehicle or any instrument mounted to or 
equipped in the vehicle. If this Is the case (Yes at S503), 
the said program is executed to carry out necessary op- 
eration under control, at S509. The result of the control 
operation is stored in the study Item data 292. 
[0197] When the applicable communication program 
is not an Instrument control program (No at S503), at 
step S504, agent processing unit detennines a display 
background, in reference to the background select table 
306 (Fig. 40). from the current status (the current time, 
season. Instruments in operation, status of driver's voice 
recognition, goal inputted in the navigation system, etc.) 
and the response recognition data 307 in agent data 
memory 29. The applicable communication is executed 
by displaying the selected background, together with the 
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prescribed agent's activity, on display 27, at S505. Tine 
agent may also talk to the driver through speaker 25 in 
accordance with the contents of the applicable commu- 
nication program. Thus, the agenf s activity (action and 
voice) such as shown in Figs. 41 A-41 D is shown on dis- 
play 27 with a suitable background that Is pertinent to the 
agent's activity. 

[0198] When this communication program is not a 
questioning or responsive one requiring the driver's any 
response (No at S506), the procedure is retumed to the 
main routine. When this Is the case (Yes at S506), agent 
processing unit 1 1 acquires the driver's response or an- 
swer through microphone 26 and voice processing unit 
14 or through the driver's operation of Input device 22. 
at step S507. The acquired response is incorporated into 
response data 293 at step S508, then the procedure Is 
retumed to the main routine. The data accumulation or 
the past record study at S508 has been described In con- 
nection with the preceding embodiments. In this embod- 
iment, when the driver's response acquired at S508 is 
any reaction to the agent's activity that is executed in 
response to the last time driver's affinnative or negative 
answer in reply to the agent's questioning activity, this 
reaction is stored in the response recognition data 298 
so that the distribution and the most recognizable answer 
are rewritten when necessary, as having been described. 
[0199] Figs. 44A-44C show examples of the agent's 
activities shown on display 27. with the pertinent back- 
ground, in the communication program executed at 
S505. Fig. 44A represents the current status data ac- 
quired at S1 1 3 of the main routine of Fig. 1 5 and contents 
of the study Item data 292 and the response data 293 In 
agent data memory 29. The acquired current status data 
includes the current time (17:00), the current vehicle po- 
sition (Tokyo) that is detected by the vehicle position sen- 
sor 21. The study item data 292 and the response data 
293 storing the past record study represents in this ex- 
ample that the driver has most frequently been listening 
to the FM radio program "J-WAVE" during 17:00-18:00. 
[0200] From these data in Fig, 44A, agent processing 
unit 1 1 selects a communication program In reference to 
the program select table 291 at S501 . In this case, the 
communication program No. 00501 in which the agent 
suggests the driver listening the radio program Is select- 
ed at S501 . This program is not subjected to suspension 
or skip, which is confinned at S502. Then, it is conf imried 
that this program Is not an Instmment control program at 

5503. Agent processing unit 1 1 then detennines a back- 
ground to be displayed together with the agent's activity 
in the selected communication program No. 00501 , at 

5504. The background selection is made in reference to 
the table 306 of Fig. 40. For instance, since the current 
time is between 1 5:00-1 8:00, a sunset background is se- 
lected. Further, this program No. 00501 involves the 
agenf s speaking, and the driver's voice spoken at the 
same with the agent's speaking would hardly be recog- 
nized. Accoixllngly, when executing the program Involv- 
ing the agenf s speaking, display 27 has a red frame in- 



dicating the voice unrecognizable situation now. Thus, a 
red frame background is also selected at S504. 
[0201 ] After one or more background has been select- 
ed in this way, the communication program No. 00501 Is 

5 executed at S505 so that the agenf s activity and the 
selected background are shown on display 27, as shown 
in Fig. 44B, More particularly, the agent appears on dis- 
play 27 with a sunset background and says "You may 
listen to the radio program?". This communication pro- 

10 gram does not need the driver's response (No at S506) 
so that the procedure is retumed to the main routine of 
Fig. 15. 

[0202] In the next application of the main routine of 
Fig. 15, the driver's answer awaiting condition created 

15 by the program No. 00501 is newly confirmed at S1 1 3 of 
Fig. 1 5 as one of the current status. As a consequence, 
another communication program is selected at S501 for 
obtaining the driver's answer in response to the agenf 
speaking (Rg. 44B). Through S502 (No) and S503 (No), 

20 at 8504, agent processing unit 1 1 detenmines the back- 
ground to be applied in executing this program, in refer- 
ence to the background select table 306 (Fig. 40). In this 
case, the cun-ent time Is within the period 1 5-1 8, the pro- 
gram to be executed requires recognition of the driver's 

25 affirmative or negative answer, and the most recogniza- 
ble answers (in the response recognition data 307 of Fig. 
42) are "Yes" and "No". Accordingly, a sunset back- 
ground, two nameboards of "YES" and "NO" as the 
agenf s belongings and a yellow frame (Indicating that 

30 only the driver's affimiative and negative answers can 
be recognized) are selected as backgrounds, which are 
shown on display 27, together with a silent agent, as 
shown in Fig. 44C, when this program Is actually execut- 
ed at S505. Since this program Is a questioning program 

35 (Yes at S506), the driver's answer is obtained at S507. 
In this example, it is supposed that the driver says "Yes" 
in response to the agenf s activity of Fig. 44C. This re- 
sponse data 293 is renewed accordingly at S508, and 
the procedure Is retumed to the main routine of Fig. 15. 

40 [0203] in the next application, the fact that the driver 
made an affimiative answer in response to the agenf s 
activity in the previously executed communication pro- 
gram Is confinned at S1 13 as one of the cunrent status. 
As a consequence, still another communication program 

45 is selected at S501 for turning the radio receiver on and 
selecting the radio channel (Fig. 44B). This program 
should not be suspended (No at 8502). Since this is an 
instrument control program (Yes at S503), the radio re- 
ceiver is controlled accordingly at S509. In execution of 

50 this program, the radio receiver is automatically tuned to 
the most frequently selected channel in the current area 
and at the current time in accordance with the study item 
data 292. Thereafter, data regarding the radio in the study 
item data 292f (Fig. 39) is rewritten to include data In the 

55 latest operation. 

[0204] Unless the driver makes any reaction to ask 
intermptlon of operation of the radio receiver by saying 
"No" or "Stop", for example, agent processing unit 1 1 
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discriminates that the driver's answer ("Yes") has been 
. correctly recognized. Thus, the response recognition da- 
ta 307 (Fig. 42) should be renewed by incorporating this 
answer at S508, and the procedure is returned to the 
main routine of Fig. 15. 

[0205] In the next application, the operating condition 
of the radio is confimned at S113 as one of the current 
status. Still another communication program may be ap- 
plied to represent the current vehicle status (the status 
of the radio, in this case) at S501 . Through S502 and 

5503, one or more pertinent baclcground is selected at 

5504. In this case, the present time is 17:00, the radio is 
in operation and any voice Is recognizable, so that a sun- 
set background with a green frame (indicating any voice 
recognizable condition) is shown on display 27 together 
with a silent agent carrying a radio/cassette, at S505. 
Since this program is not a questioning program (No at 
S506), the procedure is soon retumed to the main routine 
of Fig. 15. 

[0206] In accordance with the above-described em- 
bodiment, not only the agent but also the background 
pertinent to the current status and/or the agent* s activity 
are represented on display 27. This enables more infor- 
mation to be displayed at the same time and more mes- 
sage to be given to the driver. The agent and the bacl<- 
ground may represent different infomnation or message. 
The background may represent where and how the ve- 
hicle is driving. For example, a tree shown as a back- 
ground may be bent to the wind (to the left). In anther 
example, where the vehicle is running on an uphill or 
downhill, such a road condition may be represented as 
a background. 

[0207] In accordance with yet another embodiment of 
the present invention, even when the same communica- 
tion program is executed, the agent's appearance and/or 
activity will be changed depending on the current status 
of the vehicle, the driver and the passengers, etc. and 
as a result of the study of the cun-ent status and the driv- 
er's response, so that the user can make more friendly 
communication with the agent. 
[0208] In this embodiment, processing unit 1 (Fig. 1) 
is connected to agent data memory 29g that stores agent 
program 290, program select table 291 , study item data 
292f, response data 293, image data 294, appearance 
select table 308 and familiarity data 300, as shown in 
Fig. 45. Program 290, table 291 and data 292-294 are 
substantially identical to those stored in agent data mem- 
ory 29 of the first embodiment of Fig. 1 . Accordingly, ex- 
planation thereof Is not repeated herein. The familiarity 
data 300 has also been described in detail in reference 
to Fig. 22. The image data 294 in this embodiment stores 
a great number of appearance such as looks, features, 
wears, belongings, situation, posture, altitude, etc. to be 
applied to the agent appearing on display 27. When the 
driver selects his or her favorite character through input 
device 22 or any other input means, the selected agent 
appears on display 27 with appearance selected in ref- 
erence to the appearance select table 308. 



[0209] Fig. 46 is an example of the appearance select 
table 308. When the vehicle cun-ent status such as de- 
scribed in the left-hand columns of the table 308 is de- 
tected by a corresponding sensor of the current status 
sensor 40 (Fig. 1), a specific appearance of the agent at 
the neighborhood center column should be selected. The 
agenf s appearance may also be determined by the past 
record study stored In the study item data 292 and/or the 
response data 293, in which case such data contents 
should be described in the left-hand columns. Each ap- 
pearance selectable in the table 308 has the order of 
priority. When a plurality of appearances are applicable 
which can not be executed at the same time, one of them 
having the eariier order of priority is selected. For exam- 
ple, when the wiper is wori<ing on Christmas Eve, ap- 
pearances of "wearing a raincoat", "wearing a sweater" 
and "dressed as Santa Claus' are applicable as the 
agent* wear, in which case the appearance of "wearing 
a raincoat" is selected according to priority so that the 
agent wearing a raincoat appears on display 27. If the 
wiper is tumed off, the appearance of the next priority is 
selected so that the agent appears under the disguise of 
Santa Clause. 

[0210] The contents of the appearance select table 
308 shown in Fig. 46 is only example and of course not 
limitative. For example, it may be possible that the type 
of the vehicle is detected when the ignition switch is 
tumed on, and the agent wears a formal suit when the 
vehicle is a high-grade sedan whereas the agent wears 
a T-shirt when the vehicle is a RV (recreation vehicle) or 
SUV (sport utility vehicle). In another example, the 
agent's height changes with the fuel remainder. The 
agent may jump up when VSC or ABS becomes opera- 
tive. The agent may be inclined while excessive accel- 
eration, deceleration or lateral gravity is applied to the 
vehicle. The agent may appear with a card indicating 
television or radio channels when the driver tums the 
television or radio on. The agent may become tall or fat 
as the driver becomes more familiar to the navigation 
system. The agent's wear or other appearance may be 
changed depending on the goal. The agent may perspire 
when the room temperature is higher than the open air 
temperature by a predetemiined difference. The agent* 
shoes may be changed depending on the road condition, 
more particularly the agent wears a pair of mountaineer- 
ing or trekking boots while the vehicle is running on an 
unpaved road. The environmental data at the left-hand 
column may Include the traffic jam, rainfall, snowfall, sun- 
shine, the number of passengers, etc. The user's data 
may include the driver's occupation. For example, the 
agent wears a while overall when the driver is a doctor. 
The user's data may also include the driver's or passen- 
ger's birthday, payday, blood type, annual income, family 
members, nationality, native language, religion, favorite 
music, favorite sport, etc. Data regarding date and time 
may include anniversary, holidays, lunch time, coffee 
brake time, dinnertime, etc. which is known to the driver 
by a pertinent agent's appearance. 
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[021 1] The main routine of the agent operation of this 
embodiment Is the same as shown in Fig. 15 and, there- 
fore, description regarding the flowchart of Fig. 1 5 should 
be referred to in understanding the agent control opera- 
tion of this embodiment. In this embodiment, however, s 
the agent control operation that Is carried out at S1 1 4 of 
the main routine of Fig. 15 Is processed as shown by the 
flowchart of Fig. 47. In the flowchart of Fig. 47, steps 
S601-602 are substantially identical to S14-15 of the 
flowchart of Fig. 9, so that description of the latter steps io 
should be refenred to In understanding the S601-602 of 
this flowchart. 

[0212] After conf inning that the applicable communi- 
cation program should be executed in this operation (No 
at S602), agent processing unit 1 1 discriminates if the is 
applicable communication program Is an Instrument con- 
trol program, at step S603. The Instmment control pro- 
gram means a program to be executed to control oper- 
ation of the vehicle or any instrument mounted to or 
equipped in the vehicle. If this is the case (Yes at S603). 20 
the said program is executed to carry out necessary op- 
eration under control, at S609. The result of the control 
operation is stored in the study item data 292, 
[0213] When the applicable communication program 
is not an instrument control program (No at S603), at 25 
step S604, agent processing unit determines an agent* s 
appearance, in reference to the appearance select table 
308 (Fig. 46), from the current status (vehicle data, nav- 
igation data, environmental data, user's data, current 
date and time, result of driver's voice recognition, driver's 30 
familiarity to the agent system etc.). The applicable com- 
munication is executed by displaying the agent with the 
selected appearance, on display 27, at S605. The agent 
may also talk to the driver through speaker 25 in accord- 
ance with the contents of the applicalbe communication 35 
program. Thus, the agent's activity (action and voice) 
such as shown in Figs, 48B, 49B and 50 is shown on 
display 27 with a suitable appearance that is pertinent to 
the agent's activity or the current status. 
[0214] When this communication program is not a 40 
questioning or responsive one requiring the driver's any 
response (No at S606), the procedure is returned to the 
main routine. When this is the case (Yes at S606), agent 
processing unit 1 1 acquires the driver's response or an- 
swer through microphone 26 and voice processing unit 45 
14 or through the driver's operation of input device 22, 
at step S607. The acquired response is incorporated into 
response data 293 at step S608, then the procedure is 
returned to the main routine. In this embodiment, espe- 
cially, agent processing unit 1 1 acquires at S607 the driv- so 
efs response to the agenf s activity in the executing com- 
munication program and a time elapse from when the 
agent* activity is outputted until when the driver's re- 
sponse is confirmed. The data accumulation or the past 
record study at S608 has been described In connection ss 
with the preceding embodiments. In this embodiment, 
when the executed communication program is a ques- 
tioning or responsive program that needs the driver's re- 



sponse, the response obtained at 608 is incorporated 
into the response data 293. When the response data 293 
already has full records, the oldest record Is deleted and 
replaced by the newly obtained record. Further, the fa- 
miliarity data 300 (Fig. 22) is renewed to a new value, as 
in the manner having been described hereinbefore in 
connection with the preceding embodiments. 
[021 5] The agent's appearance and activity in this em- 
bodiment will be described hereunder in reference to 
Figs, 48A-50 by way of example. Figs. 48A-48C illus- 
trates an example of the agent control operation execut- 
ed after the ignition switch is tumed on. As shown in Fig. 
48A, agent processing unit 11 acquires, as the cun^nt 
status, a time elapse T1 from the ignition-on from a de- 
tection signal from a timer (one of sensors 40). It has also 
been conflmned that cun'ent vehicle speed is 35kg/h, the 
driver's voice is recognizable, the wiper is wori<ing, the 
current season is summer, the current driver's familiarity 
is 27, etc. Further, agent processing unit 1 1 make refer- 
ence to the response data 293 to confirm that the driver 
showed 'acceptance' to the agenf s activity In the latest 
two application of the program No. 00123. The current 
default value TO stored in the study item data 292 is 120 
minutes. Then, it has also been confirmed that the 
elapsed time T1 S TO. 

[021 6] From these data obtained and conflmned, agent 

processing unit 11 finds the applicable communication 
program No. 00123 (Yes at S601) and confimns that ap- 
plication of this program should not be suspended at this 
time (No at S602). It also confimis that this program is 
not an instrument control program (No at S603). 
[0217] Then, agent processing unit 1 1 determines the 
agent's appearance from the cun-ent status of Fig. 48 A, 
at S604. The appearance select table 308 of Fig. 46 
teaches that, in these current status, the appearances of 
"hair falls down naturally", "full face", 'wear a swlmsuif, 
and "tall" should be selected. Accordingly, when execut- 
ing the program No. 00123 at S605, a tall agent wearing 
a swimsuit (or a sportswear) stands naturally with her full 
fact, on display 27. At the same time, she talks to the 
driver "Will you have a rest?" through speaker 25. 
[0218] Since this program is a questioning program 
(Yes at 606), the driver's response is obtained at S607, 
as shown in Fig. 48C. The driver's response "neglect", 
"refusal" or "acceptance", together with date and time of 
the response, is stored In the response data 293 of the 
program No. 00123 at S608. The oldest response data 
is replaced with the newly acquired response data. When 
the driver shows "neglect", the default value increased 
by 10 minutes is applied in the next but one application 
of this program. In case of "refusal", the default value 
increased by 20 minutes is applied in the next but five 
application of this program. In case of "acceptance", the 
same default value is applied in the next application with- 
out suspension. These are the result of the past record 
study to be incorporated in the response data 293. Then, 
the procedure is retumed to the main routine of Fig. 15. 
[021 9] Fig. 49A illustrates the different current status. 
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in which case the agent's appearance such as shown in 
Fig. 49 B Is selected In reference to the table 308 of Fig. 
46. In this case, the appearance of "hair falls down nat- 
urally", "face In profile", "wear a sweater", and "short" 
should be selected. Accordingly, when executing the pro- 
gram No. 001 23 at S605, a short agent wearing a sweater 
faces in profile, on display 27, and says "Will you have 
a rest?" through speaker 25. Operation at other steps 
will be substantially the same as in the examples of Rgs. 
49A-49C. 

[0220] In the above-described embodiment, a plurality 
of selectable agent's appearance has respective image 
data stored In the image data memory section 294. How- 
ever, it may be possible that one or several image data 
Is prepared for one communication program and 
changed by image processing unit 1 3 in accordance with 
the cun-ent status. Change of the agenf s appearance 
may Include change of the agent itself. For example, the 
agent is a rabbit when the vehicle Is running at a high 
speed whereas the agent is a turtle during low-speed 
drive. When the current status Is that the driver's voice 
Is recognizable, the agent may appear on display 27 car- 
rying a nameboard of "PLEASE" or two nameboards of 
"YES" and "NO" (Fig. 50) for urging the driver to make a 
suitable response to the agent's activity In the last com- 
munication program. 



Claims 

1. A guidance device for mounting in a vehicle, said 
guidance device comprising: 

first sensor means (40) for detecting vehicle 
conditions; 

communication program storage means (29, 
29a, 29b, 29c, 29d, 29e, 29f, 29g) containing, 
stored therein, a plurality of different communi- 
cation programs individually correlated with the 
vehicle conditions, each of said communication 
programs synthesizing at least one personified 
agent and providing communication between 
the personified agent and a user in the vehicle 
by at least one of voice and action; 
study data storing means for storing study data 
(292, 293) renewable in response to entry of the 
latest cun'ent status detected by said first sensor 
means; 

communication program select means (291, 
291b) for selecting one of said communication 
programs stored in said communication pro- 
gram storing means in accordance with said 
study data stored in said study data storing 
means; 

agent control means (11) for controlling said at 
least one of voice and action of said personified 
agent in accordance with said communication 
program selected by said communication pro- 



gram selecting means; and 
agent output means (25, 27) for outputting at 
least one of said agent's voice and action deter- 
mined by said agent control means. 

5 

2. A guidance device according to claim 1 wherein said 
first sensor means detects at least one vehicle con- 
dition selected from the group consisting of a current 
vehicle position, a cun'ent vehicle speed, a cun^ent 

10 shift position, blinker operation, wiper operation, a 
coolant temperature, an engine oil temperature, an 
engine oil pressure, a passengers compartment 
temperature an ambient air temperature, ignition 
switch operation, a brake pedal depressing stroke, 

15 an accelerator pedal depressing stroke, hand brake 
operation, horn operation, belt loosening, a fuel re- 
mainder, light or lamp operation, seat belt operation, 
door opening, window opening, ABS operation, air- 
conditioner operation, distance to the preceding and 

20 succeeding vehicles and decrease in tire air pres- 
sure. 

3. a guidance device according to claim 1 or 2 wherein 
user-related data describing a specific user is Includ- 

25 ed in said vehicle conditions detected by said first 
sensor means. 

4. A guidance device according to claim 1 , 2 or 3 further 
comprising second sensor means for detecting us- 

30 er's response to the agent's voice and action output 
by said agent output means, and response storing 
means for storing user's response data detected by 
said second sensor means. 

35 5. A guidance device according to claim 4 wherein said 
study data in said study data storing means is re- 
newed in response to entry of the latest user's re- 
sponse data in said response storing means. 

40 6. A guidance device according to claim 1 , 2. 3, 4 or 5 
wherein said agent output means comprises at least 
one of display means and speaker means. 

7. A guidance device according to any one of claims 1 
45 to 6 wherein said agent output means includes 

speaker means for outputting one of multiple voice 
patterns of the personified agent, said one voice pat- 
tem being selectively detennined by said agent con- 
trol means. 

50 

8. A guidance device according to any one of claims 1 
to 7 wherein said agent output means includes dis- 
play means for displaying the personified agent as 
an animated subject in accordance with one of mul- 

55 tiple appearance pattems for said animated subject 
said one appearance pattern being selectively de- 
termined by said agent control means. 
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9. A device according to claim 8 wherein said animated 
subject Is a pictorial representation of la human, an- 
imal or robot. 

10. A guidance device according to claim 8, wherein said s 
animated subject is a human caricature. 

1 1 . A guidance device according to any one of claims 1 
to 10 further comprising agent's character select 
means for detennining a specific characters for said io 
personified agent from among a plurality of different 
characters, said agent control means determining at 
least one of the agent's voice and action in consid- 
eration of the agent's character detemnlned by said 
agent's character select means, 

12. A guidance device according to any one of claims 1 
to 1 1 further comprising familiarity storing means for 
storing data indicating user's familiarity with agent 
operation of the guidance device, said agent control 20 
means detennining at least one of the agent's voice 
and action in consideration of the agent's familiarity 
stored in said familiarity storing means. 

13. A guidance device according to any one of claims 1 25 
to 1 2 wherein said communication programs synthe- 
size a plurality of different personified agents. 

14. A guidance device according to claim 13 further com- 
prising an agent data memory containing user-relat- 30 
ed Infomiation including identification of one of said 
personified agents as a favorite of the user. 

15. A guidance device according to any one of claims 1 
to 1 4 further comprising sub-agent storing means for 35 
storing sub-agent programs, one of said sub-agent 
programs being selected in con-espondence to a 
specific communication program stored in said com- 
munication program storing means so that a sub- 
agent of the selected sub-agent program is output 40 
by said agent output means together with said per- 
sonified agent output by execution of said selected 
communication program. 

16. A guidance device according to claim 15 wherein 
said sub-agent explains the activity of said personi- 
fied agent. 

17. A guidance device according to claim 15 or 16 
wherein said sub-agent allows short-cut of operation 
executed by said selected communication program 
for said personified agent. 

18- A guidance device according to claim 15, 16, or 17, 
wherein said one sub-agent program becomes op- 
erative when data indicating user's familiarity with 
operation of the guidance device is greater than a 
predetemnined critical value. 



19. A guidance device according to any one of claims 1 
to 18 wherein said agent output means at least in- 
cludes display means, a background of said display 
means being selectively detemilned so that said per- 
sonified agent appears on said display means to- 
gether with the selected bacl^ground. 

20. A guidance device according to to any one of claims 
1 to 19 wherein said communication program store 
din said communication program storing means In- 
clude suggestion programs by which said personi- 
fied agent makes some suggestion to the user. 



Patentansprtjche 

1. FQhmngsvorrichtung zum Einbau in einem Fahr- 
zeug, wobei die Fuhrungsvomchtung aufweist: 

eine erste Erfassungselnrichtung (40) zum Er- 
fassen von Fahrzeugzustanden; 
eine Kommunikationsprogranrimspelcherein- 
richtung (29. 29a. 29b. 29c. 29d, 29e, 29f . 29g) 
zum Speichem mehrerer verschiedener Kom- 
munikationsprogramme, die mit den Fahrzeug- 
zustanden individuell korreliert sind, wobei je- 
des der Kommunikationsprogramme minde- 
stens einen personiflzierten Agenten syntheti- 
siert und eine Kommunikation zwischen dem 
personiflzierten Agenten und einem Benutzer 
im Fahrzeug durch Sprache und/oder eine Ak- 
tion bereitstellt; 

eine Analysedatenspeichereinrichtung zum 
Speichem von Analysedaten (292, 293), die In 
Antwort auf eine Eingabe des durch die erste 
Erfassungselnrichtung erfassten aktuellsten 
Zustands aktualisierbar 1st; 
eine Kommunikationsprogrammauswahlein- 
richtung (291 , 291b) zum Auswahlen eines der 
in der Kommunikationsprogrammspeichereln- 
richtung gespelcherten Kommunikationspro- 
gramme gemaB den In der Analysedatenspei- 
chereinrichtung gespelcherten Analysedaten; 
eine Agentensteuerungseinrichtung (11) zum 
45 Steuem der Sprache und/oder der Aktion des 

personiflzierten Agenten gema3 dem durch die 
Kommunikationsprogrammauswahleinrich- 
tung ausgewahiten Kommunikationsprogramm; 
und 

50 eine Agentenausgabeelnrichtung (25, 27) zum 

Ausgeben der durch die Agentensteuerungs- 
einrichtung ausgewahiten Sprache und/oder 
Aktion des Agenten. 

55 2. VoR-ichtung nach Anspruch 1 , wobei die erste Erfas- 
sungselnrichtung mindestens einen Fahrzeugzu- 
stand erfasst, der ausgewahit wird aus: einer Fahr- 
zeug-lstPosltion, einer aktuellen Fahrzeugge- 
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schwindigkeit, einer aktuellen Schaltstellung, einer 
Fahrtrichtungsanzeigerbetatigung, einer Scheiben- 
wischerbetatigung, einer Kuhlmitteltemperatur. ei- 
ner Motoroltemperatur, einem Motor6!druck, einer 
Fahrgastraumtemperatur, einer Umgebungsluft- 
temperatur, einer Zundschalterbetatigung, einem 
Bremspedalbetatigungsgrad, einem Beschleuni- 
gungspedalbetatigungsgrad, einer Handbremsen- 
betatigung, einer Aktivierung des Hupsignals, einer 
Lockerung des Sicherheitsgurts, einer Kraftstoff rest- 
menge, einer Licht- Oder Lampenaktivlerung, einer 
SichertieitsgurtbetStigung, einer TOroffnung, einer 
Fensteroffnung, einer ABS-Aktivierung, einer Akti- 
vierung der Klimaanlage, einem Abstand zu einem 
vorausfahrenden bzw. einem nachfolgenden Fahr- 
zeug und einer Abnahme des Reifendrucks. 

3. Vonichtung nacli Ansprucli 1 Oder 2, wobel in den 
durch die erste Erfassungseinrichtung erfassten 
Fahrzeugzustanden benutzerspezifisclie Daten ent- 
halten sind, die einen spezifischen Benutzer be- 
schrelben. 

4. Von-ichtung nach Anspruch 1, 2 Oder 3 femer mit 
einer zweiten Erfassungseinrichtung zum Erfassen 
einer Benutzerantwort auf die durch die Agenten- 
ausgabeeinrichtung ausgegebene Sprache und Ak- 
tion des Agenten und einer Antwortspelchereinrich- 
tung zum Speichem von durch die zweite Erfas- 
sungseinrichtung erfassten Benutzerantwortdaten. 

5. Vonichtung nach Anspruch 4. wobei die Analyseda- 
ten in der Analysedatenspeichereinrichtung in Ant- 
wort auf die Eingabe der aktuellsten Benutzerant- 
wortdaten in die Antwortspeichereinrichtung aktua- 
lisiert werden. 

6. Vonichtung nach Anspruch 1 , 2, 3, 4 oder 5, wobei 
die Agentenausgabeeinrichtung ein Display und/ 
Oder einen Lautsprecher aufwelst. 

7. Vonichtung nach einem der Anspruche 1 bis 6, wo- 
bei die Agentenausgabeeinrichtung einen Lautspre- 
cher zum Ausgeben eines von mehreren Sprachmu- 
stem des personifizierten Agenten aufweist, wobei 
das eine Sprachmuster durch die Agentensteue- 
mngseinrichtung selektiv bestimmt wird. 

8. Vonichtung nach einem der Anspruche 1 bis 7, wo- 
bei die Agentenausgabeeinrichtung ein Display zum 
Darstellen des personifizierten Agenten ais animier- 
tes Subjekt gemd6 einem von mehreren Erschei- 
nungsbildmustem fur das animierte Subjekt auf- 
weist, wobei das eine Erscheinungsbildmuster 
durch die Agentensteuerungseinrichtung selektiv 
bestimmt wird. 

9. Vomchtung nach Anspruch 8, wobei das animierte 



Subjekt eine Bilddarstellung eines Menschen, eines 
Tiers oder eines Roboters ist. 

10. Vonichtung nach Anspmch 8, wobel das animierte 
5 Subjekt eine menschliche Karikatur ist. 

11. Vorichtung nach einem der Anspruche 1 bis 10, fer- 
ner mit einer Agentencharakterauswahleinrichtung 
zum Bestimmen eines spezifischen Charakters fur 

10 den personifizierten Agenten aus mehreren ver- 
schiedenen Charaktem, wobei die Agentensteue- 
rungseinrichtung die Sprache und/oder die Aktion 
des Agenten bezuglich des durch die Agentencha- 
rakterauswahleinrichtung bestimmten Agentencha- 

15 rakters bestimmt. 

12. Von'ichtung nach einem der AnsprOche 1 bis 11 , fer- 
ner mit einer Vertrautheitsspeichereinrichtung zum 
Speichem von Daten, die die Vertrautheit des Be- 

20 nutzers mit der Agentenoperation der Fuhrungsvor- 
richtung anzeigen, wobei die Agentensteuerungs- 
einrichtung die Sprache und/oder die Aktion des 
Agenten bezuglich den in der Vertrautheltsspeicher- 
einrichtung gespeicherten Daten bestimmt. 

25 

1 3. Von-ichtung nach einem der Anspruche 1 bis 1 2, wo- 
bei die Kommunikationsprogramme mehrere ver- 
schiedene personifizierte Agenten synthetisieren. 

30 14. Vonichtung nach Anspruch 13, femer mit einem 
Agentendatenspeicher, der benutzerspezifische In- 
forrnatlon mit einer Identifizierung eines der perso- 
nifizierten Agenten als einen Favoriten des Benut- 
zers enthSlt. 

35 

15. Von-ichtung nach einem der Anspruche 1 bis14,fer- 
ner mit einer Unteragentenspeichereinrichtung zum 
Speichem von Unteragentenprogrammen, wobel ei- 
nes der Unteragentenprogramme entsprechend ei- 

40 nem spezifischen Kommunikationsprogramm, das 
in der Kommunikationsprogrammspeichereinrich- 
tung gespeichert Ist, derart ausgewahit wird, dass 
ein Unteragentdesausgewahlten Unteragentenpro- 
gramms durch die Agentenausgabeeinrichtung zu- 

45 sammen mit dem personifizierten Agenten ausge- 
geben wird, der durch Ausfuhren des ausgewahlten 
Kommunikationsprogramms ausgegeben wird. 

16. Vonichtung nach Anspruch 15, wobei der Unter- 
50 agent die AktivitSt des personifizierten Agenten er- 

Idutert. 

17. Von'Ichtung nach Anspruch 15 oder 16. wobei der 
Unteragent eine VerkOrzung der Operation ermog- 

55 licht, die durch das ausgewahlte Kommunikations- 
programm fur den personifizierten Agenten ausge- 
fuhrtwird. 
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18. Vorrichtung nach Anspruch 15. 16 Oder 17. wobei 
das Unteragentenprogramm aktivlertwird, wenn ein 
Datenelement, das die Vertrautheit des Benutzers 
mit der Operation der Fuhrungsvorrichtung anzeigt, 
groBer ist ais ein vorgegebener l<ritischer Wert. 

19. Vorrichtung nach einemder Anspruche 1 bis 18, wo- 
bei die Agentenausgabeeinrichtung mindestens ein 
Display aufweist, wobei ein Hintergrund des Dis- 
plays selektlv bestimmbar Ist, so dass der personl- 
fizierte Agent auf dem Display zusammen mit dem 
ausgewShlten Hintergrund erscheint 

20. Vorrichtung nach einem der AnsprOche 1 bis 1 9. wo- 
bei das In der Kommunikatlonsprogrammspeicher- 
einrlchtung gespeicherte Kommunikationspro- 
gramm Vorschlagprogramme aufweist, durch die 
der personifizierte Agent dem Benutzer einlge Vor- 
schlSge unterbreitet. 



Revendlcations 

1. Disposltif de guidage k monter dans un v6hicute, 
ledit dispositif de guidage comprenant: 

- un premier moyen de capteur (40) pour detec- 
tor les conditions du vehicule ; 

- un moyen de stockage de programme de com- 
munication (29. 29a, 29b. 29c, 29d, 29e, 29f, 
29g) contenant, stockes k I'int6rieur, une plura- 
lity de diff^rents programmes de communica- 
tion lies indlviduellement aux conditions du ve- 
hicule. chacun desdits programmes de commu- 
nication synthetisant au moins un agent person- 
nlfl6 et foumissant une communication entre 
I'agent personnifie et un utillsateur dans le ve- 
hicule par le biais d'au moins une voix et une 
action ; 

- un moyen de stockage des donn6es d'etude 
pour stocker les donnees d'etude (292, 293), 
renouvelable en r^ponse h Tentree du demier 
etat courant detecte par ledit premier moyen de 
capteur : 

- un moyen de selection du programme de com- 
munication (291, 291b) pour cholsir un desdits 
programmes de communication stockes dans 
ledit moyen de stockage de programme de com- 
munication, confomement auxdites donn6es 
d'etude stockees dans ledit moyen de stockage 
de donnees d'etude ; 

- le moyen de controle de I'agent (1 1 ) pour con- 
trdler au moins un element parmi la voix et Tac- 
tion dudit agent personnifie. confomnement 
audit programme de communication choisi par 
ledit moyen de selection du programme de 
communication ; et 

- un moyen d'emission d'agent (25. 27) afin 



d'emettre au moins un des elements entre la 
voix et I'action de I'agent, determine par ledit 
moyen de controle de I'agent. 

5 2. Dispositif de guidage selon la revendication 1 , dans 
iequel ledit premier moyen de capteur detecte au 
moins une condition de vehicule choisie dans le 
groupe consistant en une position actuelle du vehi- 
cule. une Vitesse actuelle du vehicule, une position 
10 de changement actuelle. le fonctionnement du cil- 
gnotant. le fonctionnement de Tessule-glace, la tem- 
perature du liquide de refroidissement, la tempera- 
ture de I'huile moteur, la presslon d'huile moteur, la 
temperature du compartiment passager, la tempe- 
rs rature de I'air ambient, le fonctionnement du com- 
mutateur d'allumage, la course de depression de la 
pedale de f rein, la course de depression de la pedale 
d'acceierateur, le fonctionnement du frein k main, le 
fonctionnement du klaxon, le desserrage de la cour- 
20 roie, le reste d'essence, le fonctionnement des feux 
ou phares, le fonctionnement de la ceinture de se- 
curite, I'ouverture de porte, I'ouverture de fenetre, le 
fonctionnement de TABS, le fonctionnement du cli- 
matlseur, la distance entre les vehlcules precedents 
25 et suivants et la diminution de la presslon d'air du 
pneu. 

3. Dispositif de guidage selon la revendication 1 ou 2. 
dans lequei les donnees liees k Putilisateur decrivant 

30, un utillsateur specifique sont comprises dans lesdi- 
tes conditions du vehicule detectees par ledit pre- 
mier moyen de capteur. 

4. Dispositif de guidage selon la revendication 1 . 2 ou 
35 3, comprenant en outre un second moyen de capteur 

pour detecter la reponse de Tutilisateur k la voix et 
k Taction de Tagent, emise par ledit moyen d'emis- 
sion d'agent, et un moyen de stockage de reponse 
pour stocker les donnees de reponse de Tutilisateur 
40 detecte par ledit second moyen de capteur. 

. 5. Dispositif de guidage selon la revendication 4, dans 
lequei lesdites donnees d'etude dans ledit moyen 
de stockage des donnees d'etude sont renouveiees 
45 en reponse k Tentree de la demiere donnee de re- 
ponse de Tutilisateur dans ledit moyen de stockage 
de la reponse. 

6. Dispositif de guidage selon la revendication 1 , 2. 3, 
50 4 ou 5. dans lequei ledit moyen d'emission d'agent 

comprend au moins un element panni un moyen 
d'affichage et un systeme de haut parleur. 

7. Dispositif de guidage selon Tune quelconque des re- 
55 vendications 1 k 6. dans lequei ledit moyen d'emis- 
sion d'agent comprend un systfeme de haut-parleur 
pour emettre un des multiples modules de voix de 
Tagent personnifie, ledit un modeie de voix etant se- 
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lectivement d6termin6 par (edit moyen de contrdle 
de Tagent. 

8. Dispositif de guidage selon Tune quelconque des re- 
vendications 1 k 7, dans lequel ledit moyen d'§mls- 
slon de I'agent comprend un moyen d'affichage pour 
afficher i'agent personnifi^ sous la fomie d'un sujet 
anim^, confoim^ment k un des multiples modules 
d'apparence pour ledit sujet anim^, ledit module 
d'apparence 6tant s^lectivement d^tenmin^ par ledit 
moyen de contrdle d'agent. 

9. Dispositif selon la revendicatlon 8, dans lequel ledit 
sujet anim6 est une representation illustrSe d'un §tre 
humain, d'un animal ou d'un robot. 

10. Dispositif de guidage selon la revendicatlon 8, dans 
lequel ledit sujet anim6 est une caricature humaine. 

1 1 . Dispositif de guidage selon i'une quelconque des re- 
vendications 1^10, comprenant un moyen de se- 
lection du caract^re de I'agent, afin de determiner 
un caractere sp6cifique pour ledit agent personnifl6 
pamii une plurality de differents caract^res, ledit 
moyen de contrdle de i'agent detemiinant au moins 
un element parmi la voix et Taction de I'agent en 
tenant compte du caractere de I'agent detennine par 
ledit moyen de selection du caract^re de I'agent. 

12. Dispositif de guidage selon I'une quelconque des re- 
vendications 1 k 11, comprenant en outre un moyen 
de stockage de f amiliarite pour stocker des donnees 
indiquant la familiarite de I'utilisateur avec le fonc- 
tionnement de I'agent du dispositif de guidage, ledit 
moyen de contrdle de i'agent detenminant au moins 
un element parmi la voix et Taction de I'agent. en 
tenant compte de la familiarite de I'agent stockee 
dans ledit moyen de stockage de familiarite. 

1 3. Dispositif de guidage selon I'une quelconque des re- 
vendications 1^12, dans lequel lesdits programmes 
de communication synthetlsent une pluralite de dif-. 
ferents agents personnifies. 

1 4. Dispositif de guidage selon la revendicatlon 1 3 com- 
prenant en outre une memoire de donnees d'agent 
contenant des informations liees k I'utilisateur com- 
prenant I'ldentification d'un desdits agents person- 
nifies comme un favor! de rutillsateur. 

1 5. Dispositif de guidage selon I'une quelconque des re- 
vendications 1 di 14. comprenant en outre des 
moyens de stockage de sous-agent, pour stocker 
les programmes sous-agents, Tun desdits program- 
mes sous-agents etant choisi en correspondence 
avec un programme de communication speclflque 
stocke dans ledit moyen de stockage de programme 
de communication, de sorte qu'un sous-agent du 



programme de sous-agent seiectionne soit emis par 
ledit moyen d'emlssion d'agent avec ledit agent per- 
sonnifie emis par {'execution dudit programme de 
communication choisi. 

5 

1 6. Dispositif de guidage selon la revendicatlon 1 5, dans 
lequel ledit sous-agent explique I'actlvite dudit agent 
personnlfie. 

10 17. Dispositif de guidage selon la revendicatlon 15 ou 
16, dans lequel ledit sous-agent pemnet un raccourcl 
du fonctronnement execute par ledit programme de 
communication choisi pour ledit agent personnifie. 

15 18. Dispositif de guidage selon la revendicatlon 15, 16 
ou 17, dans lequel ledit programme sous-agent de- 
vient operationnel quand les donnees Indiquant la 
familiarite de I'utilisateur avec le fonctionnement du 
dispositif de guidage sont superieures k une valeur 

20 critique predeterminee. 

1 9. Dispositif de guidage selon I'une quelconque des re- 
vendications 1^18, dans lequel ledit moyen d'emls- 
sion d'agent comprend au moins un moyen d'affi- 
25 chage, un arriere-plan dudit moyen d'affichage etant 
seiectivement detemiin6, de sorte que ledit agent 
personnifie apparaisse sur ledit moyen d'affichage 
avec raniere-plan choisi. 

30 20. Dispositif de guidage selon I'une quelconque des re- 
vendications 1^19, dans lequel ledit programme de 
communication stocke dans ledit moyen de stocka- 
ge de programme de communication comprend des 
programmes de suggestion, par lesquels ledit agent 

35 personnifie fait queiques suggestions k I'utilisateur. 
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FIG. 3 



Program 
No. 


Aeenfs Activiries 


Action 


Utterance 


OOOOl 


Obeisance 


•"Nice to meet vou, Vm Lisa. Mav I bo with vou ?" 


00002 


Obeisance 


"Nice lo meet vou. I'm Lisa, Mav I eo with vou ?" 


00003 


Obeisance 


"Hello. Vm Lisa. I will eo with vou aeaia Did vou sleeo well last nieht ?** 


00004 


Obeisance 


"Hello. Tra Lisa. I will eo with vou aeain." 


00005 


Obeisance 


"What haooens with vou ?" 


00006 


Obeisance 


"Did vou have a good rest ?** 


00007 


Bow 


"Good Riomine. Vm Lisa Mav i so with vou ?" 


• 


Bow • 


"Good afternoon. Vm Lisa. Mav 1 eo with vou 7* 


Bow 


"Good evenine. Vm Lisa. Mav I eo \vith vou ?" 


Bow 


"Good momine. Vm Lisa. Mav I eo with vou ?" 


Bow 


"Good afternoon. I'm Lisa. Mav I EO wilh vou ?" 


Bow 


"Good evenine. I'm Lisa. Mav I eo with vou ?" 


Bow 


"Hello. Vm Lisa, I will eo with vou aeain. Did vou sleep well last nieht ?" 


Bow 


"Hello. I'm Lisa. I will eo with vou aeaia Rest vourself " 
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"Good momine. Vm Lisa. It's been a lone time, Mav I eo with vou ?" 
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"Hi 1 Go with me aeain cheerfullv. Did vou have a eood rest 7" 


Cheerful 


"Rest vounelf. Go with me aeain cheerfullv. " 
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"Good evenine 1 It's been a lone time. Go with me aeain cheerfullv. " 


Doubtful 


"What haooens with vou 7" 
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"Good moraine 1 It's been a lone lime. Go with me aeain cheerfully. " 
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"Good afternoon 1 It's been a lone time. Go with me aeain cheerfullv. " 


Sleepv 


"Good evenine ! It's been a lone time. Go with me aeain cheerfullv." 
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